Display device, display method, and computer-readable recording medium

ABSTRACT

A display device includes a circuitry configured to perform a search for a plurality of image candidates in an image transformation dictionary part, based on handwritten data, and a display configured to display the plurality of image candidates obtained by the search. At least a portion of the plurality of image candidates displayed on the display represents a different person or an object.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is based upon and claims priority to JapanesePatent Applications No. 2019-227591, filed on Dec. 17, 2019, and No.2020-200247, filed on Dec. 2, 2020, the entire contents of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to a display device, a display method,and a computer-readable recording medium.

2. Description of the Related Art

Display devices for displaying data handwritten on a touchscreen panelusing a pen or finger are known. The display devices having a relativelylarge touchscreen panel may be arranged in a conference room or thelike, and may be used by a plurality of users, as the electronicwhiteboard or the like.

In some display devices, the handwritten data may be recognized andconverted into text (Japanese Kana and Kanji characters), as describedin Japanese Unexamined Patent Application Publication No. H06-251203,for example. According to the technique proposed in Japanese UnexaminedPatent Application Publication No. H06-251203, the handwritten inputcharacter string is registered in a personal dictionary, incorrespondence with a result of a conversion process performed on theinput character string in the past using a Kana Kanji conversiondictionary. When the handwritten character string is input from atablet, the personal dictionary is searched for the input characterstring, before performing the conversion process using the Kana Kanjiconversion dictionary.

However, the conventional display devices can only display characters,as the selectable candidates. In general, the display devices displaystrokes that are handwritten by the user, as they are, or displays theconversion candidates of the characters based on the strokes. Hence,whether or not the displayed strokes or characters are appropriate, canonly be judged from the characters. However, in the case of long terms,easily mistaken terms, terms having similar terms, or technical terms,the user may erroneously select unintended characters. When thecandidate is erroneously selected, the display device may display thecharacters that are different from those of the person or objectintended by the user.

SUMMARY

According to one aspect of the embodiments, a display device includes acircuitry configured to perform a search for a plurality of imagecandidates in an image transformation dictionary part, based onhandwritten data; and a display configured to display the plurality ofimage candidates obtained by the search, wherein at least a portion ofthe plurality of image candidates displayed on the display represents adifferent person or an object.

Other features of the embodiments will be apparent from the followingdetailed description when read in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for explaining an example of a character display notintended by a user.

FIG. 2 is a diagram illustrating an example of a plurality of imagecandidates displayed on a display device.

FIG. 3 is a perspective view illustrating an example of a pen.

FIG. 4A, FIG. 4B, FIG. 4C, and FIG. 4D are diagrams illustrating anexample of an overall configuration of the display device.

FIG. 5 is a diagram illustrating an example of a hardware configurationof the display device.

FIG. 6A and FIG. 6B are diagrams for explaining functions of the displaydevice and the pen.

FIG. 7 is a diagram illustrating an example of defined control data.

FIG. 8 is a diagram illustrating an example of dictionary data of ahandwriting recognition dictionary part.

FIG. 9 is a diagram illustrating an example of dictionary data of acharacter string conversion dictionary part.

FIG. 10 is a diagram illustrating an example of dictionary data of apredicted conversion dictionary part.

FIG. 11 is a diagram illustrating an example of image dictionary data ofan image transformation dictionary part.

FIG. 12A and FIG. 12B are diagrams illustrating an example of operationcommand definition data and system definition data stored in anoperation command definition part.

FIG. 13 is a diagram illustrating an example of the operation commanddefinition data when a selected object selected by a handwritten objectis present.

FIG. 14 is a diagram illustrating an example of user definition datastored in the operation command definition part.

FIG. 15 is a diagram illustrating an example of handwritten signaturedata stored in a handwritten signature data storage part.

FIG. 16 is a diagram illustrating an example of handwritten inputstorage data stored in a handwritten input storage part.

FIG. 17A and FIG. 17B are diagrams for explaining pen ID control datastored in a pen ID control data storage part.

FIG. 18 is a diagram illustrating an example of an operation guide andselectable candidates displayed by the operation guide.

FIG. 19 is a diagram (part 1) illustrating an example of the operationguide in which only image candidates are displayed.

FIG. 20 is a diagram (part 2) illustrating an example of the operationguide in which only the image candidates are displayed.

FIG. 21 is a diagram (part 3) illustrating an example of the operationguide in which only the image candidates are displayed.

FIG. 22 is a diagram (part 1) illustrating a display example of theoperation guide displaying the image candidates and the names of theimage candidates.

FIG. 23 is a diagram (part 2) illustrating a display example of theoperation guide displaying the image candidates and the names of theimage candidates.

FIG. 24 is a diagram (part 3) illustrating a display example of theoperation guide displaying the image candidates and the names of theimage candidates.

FIG. 25A, FIG. 25B, FIG. 25C, and FIG. 25D are diagrams for explainingan example of specifying the selected object.

FIG. 26A and FIG. 26B are diagrams illustrating a display example ofcandidates of operation commands based on the operation commanddefinition data when the handwritten object is present.

FIG. 27A and FIG. 27B are diagram illustrating a display example of thecandidates of the operation commands based on the operation commanddefinition data when the handwritten object is present.

FIG. 28A, FIG. 28B, and FIG. 28C are diagrams for explaining a method ofinputting angular information of 90 degrees.

FIG. 29 is a diagram for explaining another method of inputting theangular information.

FIG. 30A, FIG. 30B, and FIG. 30C are diagrams for explaining a method ofregistering the handwritten signature data.

FIG. 31 is a diagram illustrating an example of the operation guide thatis displayed when a user handwrites “Suzuki” which is a registeredhandwritten signature data.

FIG. 32A and FIG. 32B are diagrams for explaining a method of changingthe user definition data.

FIG. 33 is a sequence diagram (part 1) for explaining an example of aprocess in which a display device displays character string candidatesand operation command candidates.

FIG. 34 is a sequence diagram (part 2) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 35 is a sequence diagram (part 3) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 36 is a sequence diagram (part 4) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 37 is a sequence diagram (part 5) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 38 is a sequence diagram (part 6) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 39 is a sequence diagram (part 7) for explaining the example of theprocess in which the display device displays the character stringcandidates and the operation command candidates.

FIG. 40 is a diagram illustrating an example of the operation guidedisplaying the image candidates of pharmaceutical products.

FIG. 41 is a diagram illustrating an example of the operation guidedisplaying the image candidates of portrait photographs.

FIG. 42 is a diagram illustrating an example of the operation guidedisplaying molecular structure models as the image candidates.

FIG. 43 is a diagram (part 1) illustrating an example of the operationguide displaying the image candidates of the molecular structure modelin 3-Dimensional Computer Graphics (3DCG).

FIG. 44 is a diagram (part 2) illustrating an example of the operationguide displaying the image candidates of the molecular structure modelin 3DCG.

FIG. 45 is a diagram (part 3) illustrating an example of the operationguide displaying the image candidates of the molecular structure modelin 3DCG.

FIG. 46 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the pharmaceutical products.

FIG. 47 is a diagram for explaining a configuration example of an imagetransformation dictionary part.

FIG. 48 is a diagram illustrating an example of the image transformationdictionary part including handwritten data.

FIG. 49 is a diagram illustrating an example of a plurality of imagecandidates displayed by the display device when making an Englishconversion.

FIG. 50 is a diagram illustrating an example of dictionary data of thehandwriting recognition dictionary part used for the English conversion.

FIG. 51 is a diagram illustrating an example of the dictionary data ofthe character string conversion dictionary part used for the Englishconversion.

FIG. 52 illustrates an example of the dictionary data of the predictedconversion dictionary part used for the English conversion.

FIG. 53 illustrates an example of the image dictionary data included inthe image transformation dictionary part used for the Englishconversion.

FIG. 54A and FIG. 54B are diagrams illustrating an example of theoperation command definition data for a case where no selected object ispresent when performing the English conversion.

FIG. 55 is a diagram illustrating an example of the operation commanddefinition data for a case where a selected object is present whenperforming the English conversion.

FIG. 56 is a diagram illustrating an example of the user definition dataused for the English conversion.

FIG. 57 is a diagram illustrating an example of the operation guide andthe selectable candidates displayed by the operation guide whenperforming the English conversion.

FIG. 58 is a diagram illustrating an example of a transition state inwhich the image candidates are gradually narrowed down according to theuser's handwriting when performing the English conversion.

FIG. 59 is a diagram illustrating an example of the transition state inwhich the image candidates are gradually narrowed down according to theuser's handwriting when performing the English conversion.

FIG. 60 is a diagram illustrating an example of the transition state inwhich the image candidates are gradually narrowed down according to theuser's handwriting when performing the English conversion.

FIG. 61 is a diagram illustrating a display example of the imagecandidates and the operation guide displaying names of the imagecandidates when performing the English conversion.

FIG. 62 is a diagram illustrating a display example of the imagecandidates and the operation guide displaying the names of the imagecandidates when performing the English conversion.

FIG. 63 is a diagram illustrating a display example of the imagecandidates and the operation guide displaying the names of the imagecandidates when performing the English conversion.

FIG. 64A and FIG. 64B are diagrams for explaining a specifying exampleof the selected object when performing the English conversion.

FIG. 65A and FIG. 65B are diagrams illustrating a display example of theoperation command candidates based on the operation command definitiondata when the handwritten object illustrated in FIG. 64A and FIG. 64B ispresent, respectively.

FIG. 66A, FIG. 66B, and FIG. 66C are diagrams for explaining an exampleof a method of inputting the angular information when performing theEnglish conversion.

FIG. 67A, FIG. 67B, and FIG. 67C are diagrams illustrating a method ofregistering a handwritten signature data in the case of an Englishconversion.

FIG. 68 is a diagram illustrating an example of the operation guidedisplayed in a case where “Bob”, which is the registered handwrittensignature data, is handwritten by the user when performing the Englishconversion.

FIG. 69A and FIG. 69B are diagrams for explaining a method of changingthe user definition data when performing the English conversion.

FIG. 70 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the pharmaceutical products.

FIG. 71 is a diagram illustrating an example of the operation guidedisplaying the image candidates of portrait photographs when performingthe English conversion.

FIG. 72 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the molecular structure models whenperforming the English conversion.

FIG. 73 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the molecular structure models in3DCG when performing the English conversion.

FIG. 74 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the molecular structure models in3DCG when performing the English conversion.

FIG. 75 is a diagram illustrating an example of the operation guidedisplaying the image candidates of the molecular structure models in3DCG when performing the English conversion.

FIG. 76 is a diagram illustrating an example of the operation guidedisplaying the image candidates of pharmaceutical products whenperforming the English conversion.

FIG. 77 is a diagram for explaining an association of images in theimage transformation dictionary part.

FIG. 78 is a diagram illustrating an example of the image transformationdictionary part searched using the handwritten data.

FIG. 79 is a diagram illustrating another configuration example of thedisplay device.

FIG. 80 is a diagram illustrating still another configuration example ofthe display device.

FIG. 81 is a diagram illustrating a further configuration example of thedisplay device.

FIG. 82 is a diagram illustrating another configuration example of thedisplay device.

DESCRIPTION OF THE EMBODIMENTS

Embodiments will hereinafter be described with reference to thedrawings. In drawings, the same constituent elements are designated bythe same reference numerals, and a repeated description of the sameconstituent elements may be omitted.

One object of the embodiments is to provide a display device capable ofdisplaying information other than characters, as the selectablecandidates.

Hereinafter, a display device, a display method employed by the displaydevice, and a computer-readable recording medium according to theembodiments of the present invention will be described with reference tothe drawings.

First Embodiment

<Utilized Scenes>

The display device may be utilized in various scenes or situations,including conferences, for example, and displays information accordingto the scene. For example, a researcher in industry or local governmentmay handwrite treatments using chemicals, and a physician in medicalpractice may handwrite drugs to be prescribed and present the drugs tonurses. In addition, in school, a teacher may handwrite particularexamples of things that will be taught in class, while a presenter inindustry may handwrite product names and concepts so as to introduce theproducts.

In the Japanese language, there are Hiragana characters, Katakanacharacters, and Kanji characters, instead of alphabets. A Japanese wordor term may be spelled by one or more Hiragana characters, Katakanacharacters, Kanji characters, or a combination of at least two of suchJapanese characters (hereinafter also simply referred to as “characters”or “character string” unless otherwise indicated). Further, Japanesetext data may have one of two orientations, and the Japanese charactersmay be written in a horizontal direction from left to right, or in avertical direction from top to bottom.

However, the user may handwrite or select characters that differ fromthe characters indicating the intended person or object. FIG. 1 is adiagram for explaining an example of a character display not intended bythe user. For example, the user may handwrite the Katakana character “

” pronounced “

” and illustrated at the top left in FIG. 1, which is also illustratedas a handwritten object 504 within a handwritten object rectangular areadisplay 503 illustrated at the top right in FIG. 1, with the intentionof selecting the Japanese term for “ascorbic acid”, but may erroneouslyselect the Japanese term for “aspartic acid” from among the characterstring candidates because the first two characters and the last twocharacters for these two Japanese terms are the same and these twoJapanese terms look similar at first glance. Because the Japanese termfor “ascorbic acid” is also displayed as the candidate in the exampleillustrated in FIG. 1, the user is less likely to erroneously select theJapanese term for “aspartic acid”, but if the Japanese term for“ascorbic acid” is not displayed as one of the candidates, the user ismore likely to erroneously select a similar Japanese term.

In addition, at conferences or the like, for example, the user mayerroneously handwrite a product name that differs from the intendedproduct name. In such a case, because the intended product name is notdisplayed as the selectable candidate to begin with, the user is morelikely to erroneously select the product name that is different from theintended product name, from the selectable candidates having productnames similar to the intended product name as in the example illustratedin FIG. 1.

As described above, when only the characters are displayed as theselectable candidates, it may be difficult for the user to notice thatthe user has selected the person or object other than the intendedperson or object, regardless of whether or not the intended term isdisplayed as the selectable candidate.

Overview of Display Device According to Embodiment

Accordingly, a display device 2 according to this embodiment, which willbe described later in conjunction with FIG. 4A through FIG. 4C, displaysnot only the characters, but also images, such as photographs,illustrations, or the like, as the selectable candidates for the termthat is being handwritten by the user. In this case, the user can moreeasily select the intended information, and reduces the possibility ofthe user erroneously selecting information that is different from theintended information of the person or object. Hence, the display device2 can display the correct information.

FIG. 2 is a diagram illustrating an example of a plurality of imagecandidates displayed on the display device 2. FIG. 2 illustrates imagesof molecular structure models as an example.

In FIG. 2, the user handwrites the Katakana character “

” pronounced “

” and illustrated at the top left in FIG. 1, which is the first Katakanacharacter of the intended Japanese term “

” for “aspartic acid”, in step (1).

In response to the handwritten first Katakana character, the displaydevice 2 displays image candidates 531 of the Japanese terms that beginwith this first Katakana character, in step (2). That is, even while theuser is handwriting this first Katakana character, the display device 2displays the image candidates 531, such as graphics, illustrations,photographs, or the like that fit a character string at this stage ofthe handwriting. FIG. 2 illustrates the example in which the molecularstructure models of ascorbic acid, aspartic acid, acetylene,acetaldehyde, and alanine are displayed. Hence, the plurality of imagecandidates 531 displayed on an operation guide 500 represent differentpersons or objects.

When the user selects one of the image candidates 531, the displaydisplays a selected image 1001 together with the characters representingthe person or object of the selected image, in step (3). In thisexample, the selected image 1001 represents the molecular structuremodel of the intended aspartic acid, and the Japanese term for thisintended aspartic acid is also displayed by a combination (or characterstring) of Katakana and Kanji characters. Alternatively, the displaydevice 2 may display only the selected image, or only the charactersrepresenting the selected image.

Because the display device 2 according to this embodiment displays theimages of the selectable candidates, it is easier for the user to selectthe intended person or object, and reduce incorrect information that isdisplayed. In addition, because the selected image can also bedisplayed, it is possible to display the information in a visuallyunderstandable manner.

Moreover, in a case where the display device 2 displays similar imagesas the selectable candidates, there is a possibility for the user toerroneously select the image of the person or object not intended by theuser. In this case, the display device 2 displays at least the imagecandidates 531 as illustrated in FIG. 2, so as to reduce the possibilityfor the user to erroneously select the image candidate 531 of the personor object not intended by the user.

It is possible to connect a Personal Computer (PC) to the display device2, and display the images to be displayed by the PC on the displaydevice 2, but this arrangement may be troublesome or time-consuming. Onthe other hand, according to this embodiment, the image candidates caneasily be displayed while arbitrary characters are being handwritten,and when the image is displayed after the image candidate is selected,it is possible to display the information in a more visuallyunderstandable manner compared to the case where only the characters aredisplayed.

Terminologies

An input device (or input means) may encompass any device (or means)that enables designating coordinates and handwriting on a touchscreenpanel. Examples of the input device include pens, user's fingers orhands, rod-shaped members, or the like. In addition, the input devicemay enable input by eye tracking using the user's line-of-sight.Handwritten data may encompass data of a locus of a series coordinatepoints that are displayed when the user continuously moves the inputdevice on the touchscreen panel. A stroke may encompass a series ofoperations in which the user presses the input device (or input means)against the touchscreen panel, continuously moves the input device inthe pressed state, and disengages and separates the input device orinput means) from the touchscreen panel. The data handwritten by thestroke may be referred to as stroke data. The handwritten data mayinclude one or more stroke data. Handwritten input indicates the inputof the handwritten data by the user.

The image candidates may encompass images of selectable candidates. Theimage is a generic term for computer-generated figures, photographs, orthe like, and may also be simply referred to as an image or graphics.Simply put, the image candidates are non-text information other thancharacters.

Representing a different person or object may encompass cases where aplurality of images relate to photographs of different people or theappearance of different products. Constituent elements of a substanceinclude molecules and atoms, but nucleons, quanta, or the likeultimately constituting these elements are referred to as particles ofmatter. In a case where the matter has a molecular structure, thisembodiment is particularly useful when each image represents an isomer.Although isomers have the same molecular formula, bonding states andspatial arrangements of the atoms differ among the isomers, and thus,the isomers are compounds exhibiting different properties.

<Example of Appearance of Pen>

FIG. 3 illustrates a perspective view of an example of a pen 2500. Inthe example illustrated in FIG. 3, the pen 2500 is a multi-function pen.The pen 2500, which has a built-in power supply and is capable oftransmitting commands to the display device 2, may be referred to as anactive pen, as opposite to a pen having no built-in power supply, whichmay be referred to as a passive pen. The pen 2500 illustrated in FIG. 3has one physical switch on a pen tip (or working end) thereof, onephysical switch on a pen tail thereof, and two physical switches on aside surface thereof. The pen tip of the pen 2500 is allocated forwriting, the pen tail of the pen 2500 is allocated for deleting, and theside surface of the pen 2500 is allocated for user functions. In thisembodiment, the pen 2500 further includes a non-volatile memory thatstores a pen ID that is unique to the pen 2500 and different from thepen IDs of other pens.

Operating procedures of the display device 2 to be performed by the usercan be reduced by using the pen with switches. The pen with switchesmainly refer to the active pens. However, passive pens having nobuilt-in power supply can generate power using only an LC circuitaccording to electromagnetic induction, and thus, the active pens mayencompass the electromagnetic induction type passive pens. Otherexamples of the pen with switches, other than the electromagneticinduction type passive pens, include optical type pens, infrared typepens, electrostatic capacitance type pens, or the like.

A hardware configuration of the pen 2500 may be similar to that of a penwhich includes a communication function and a microcomputer and employsa general control method. The pen 2500 may be an electromagneticinduction type, an active electrostatic coupling type, or the like. Inaddition, the pen 2500 may include functions such as a pen pressuredetection function, a pen tilt detection function, a pen hover functionthat displays a cursor before the pen touches the touchscreen panel, orthe like.

<Overall Configuration of Display Device>

An overall configuration of the display device 2 according to thisembodiment will be described with reference to FIG. 4A through FIG. 4D.FIG. 4A through FIG. 4C are diagrams illustrating the overallconfigurations of the display device 2, and FIG. 4D is a diagramillustrating a user U holding the pen 2500. FIG. 4A illustrates anexample of the display device 2 which is used as an electronicwhiteboard having a horizontally elongated shape and hanging on a wall.

As illustrated in FIG. 4A, a display 220 is provided at an upper portionof the display device 2. The user U, illustrated in FIG. 4D, canhandwrite (also referred to as input or draw) characters or the like onthe display 220 using the pen 2500.

FIG. 4B illustrates an example of the display device 2 which is used asan electronic whiteboard having a vertically elongated shape and hangingon the wall.

FIG. 4C illustrates an example of the display device 2 which is placedflat on a desk 230. Because the display device 2 has a thickness ofapproximately 1 cm, it is unnecessary to adjust the height of the desk230 even if the display device 2 is placed flat on the desk 230, whichmay be an ordinary or general-purpose desk. In this example, the user Ucan easily move around the desk 230.

<Hardware Configuration of Display Device>

Next, a hardware configuration of the display device 2 will be describedwith reference to FIG. 5. The display device 2 may have theconfiguration of an information processing device or a computer, asillustrated in FIG. 5. FIG. 5 is a diagram illustrating an example ofthe hardware configuration of the display device 2. As illustrated inFIG. 5, the display device 2 includes a Central Processing Unit (CPU)201, a Read Only Memory (ROM) 202, a Random Access Memory (RAM) 203, anda Solid State Drive (SSD) 204.

The CPU 201 of the display device 2 controls the overall operation ofthe display device 2. The ROM 202 stores one or more programs used todrive the CPU 201, such as an Initial Program Loader (IPL) or the like.The RAM 203 is used as a work area of the CPU 201. The SSD 204 storesvarious data, and one or more programs for the display device 2. Ofcourse, the ROM 202 and the RAM 203 may store various data.

The one or more programs may be stored in a suitable non-transitorycomputer-readable recording medium. A recording medium forming thenon-transitory computer-readable recording medium is not particularlylimited, and may include the ROM 202, the RAM 203, the SSD 204, or thelike described above.

The display device 2 further includes a display controller 213, a touchsensor controller 215, a touch sensor 216, the display 220, a powerswitch 227, a tilt sensor 217, a serial interface 218, a speaker 219, amicrophone 221, a wireless communication device 222, an infraredinterface (I/F) 223, a power control circuit 224, an AC adapter 225, anda battery 226.

The display controller 213 controls and manages screen display foroutputting an output image to the display 220 or the like. The touchsensor 216 detects a touch of an object, such as the pen 2500, theuser's hand, or the like (that is, the input device) on the display 220,that is, the contact between the input device and the display 220. Thetouch sensor 216 also receives the pen ID from the pen 2500 upondetecting the touch of the pen 2500.

The touch sensor controller 215 controls processes of the touch sensor216. The processes of the touch sensor 216 include inputting coordinatesand detecting the coordinates. The method of inputting the coordinatesand detecting the coordinates may be an optical method, for example, inthe case of the optical type touch sensor 216. According to the opticalmethod, two light emitting and receiving devices provided located atboth ends on an upper side of the display 220 emit a plurality ofinfrared rays parallel to the display 220 from respective light emittingelements, and receive, by respective light receiving elements, theinfrared rays reflected by a reflecting member provided in a peripheryof the display 220 and returned via optical paths identical to those ofthe infrared rays emitted by the respective light emitting elements. Thetouch sensor 216 outputs position information of the infrared raysemitted by the two light emitting and receiving devices and blocked bythe object, to the touch sensor controller 215, and the touch sensorcontroller 215 identifies the coordinate position, that is, a contactposition where the object makes contact with the display 220. Inaddition, the touch sensor controller 215 includes a communication unit215 a, and is capable of making wireless communication with the pen2500. A commercial pen may be used as the pen 2500 when making thecommunication according to a standard such as Bluetooth (registeredtrademark), for example. When one or more pens 2500 are preregistered inthe communication unit 215 a, the communication can be performed withoutrequiring the user to make the connection setting for enabling the pen2500 to communicate with the display device 2.

The power switch 227 turns the power of the display device 2 ON or OFF.The tilt sensor 217 detects a tilt angle of the display device 2. Thetilt sensor 217 is mainly used to detect whether the display device 2 isused in the set-up state illustrated in FIG. 4A, FIG. 4B, or FIG. 4C,and a thickness of the characters or the like may be changedautomatically according to the set-up state.

The serial interface 218 forms a communication interface with respect toan external Universal Serial Bus (USB) or the like. The serial interface218 is used to input external information, for example. The speaker 219is used for audio output, and the microphone 221 is used for audioinput. The wireless communication device 222 communicates with aterminal carried by the user, and relays a connection to the Internet,for example. The wireless communication device 222 may communicate via astandard such as Wi-Fi (registered trademark), Bluetooth (registeredtrademark), or the like, but the communication standard employed by thewireless communication device 222 is not particularly limited. Thewireless communication device 222 forms an access point, and aconnection can be made to the access point when the user sets a ServiceSet Identifier (SSID) and a password that are acquired to the terminalcarried by the user.

The following two access points (a) and (b) can be prepared for thewireless communication device 222.

Access point (a)→Internet

Access point (b)→Company network→Internet

The access point (a) may be for external users who cannot access theinternal network, but can utilize the Internet. The access point (b) isfor company users who can utilize the company (or internal) network andthe Internet.

The infrared I/F 223 detects a display device 2 arranged adjacentthereto. The infrared I/F 223 can detect only the display device 2arranged adjacent thereto by taking advantage of the linearity of theinfrared ray. One infrared I/F 223 can be provided on each side of thedisplay device 2, so that it is possible to detect the directions inwhich other display devices 2 are arranged adjacent to the displaydevice 2. The adjacent display device 2 may display handwritteninformation (handwritten information of another page when an area of onedisplay 220 is regarded as one page) that is handwritten in the past.

The power control circuit 224 controls the AC adapter 225 and thebattery 226, which are power supplies for the display device 2. The ACadapter 225 converts the alternating current (AC) shared by thecommercial power supply into direct current (DC).

In a case where the display 220 is the so-called electronic paper, thedisplay 220 consumes little or no power to maintain the image after theimage is rendered, and thus, the display 220 may be driven by thebattery 226. Accordingly, it is possible to use the display device 2 foran application such as digital signage even at a location, such asoutdoors, where a connection to the power supply is difficult.

The display device 2 further includes a bus line 210. The bus line 210may be an address bus, a data bus, or the like for electricallyconnecting each of the constituent elements of the display device 2,such as the CPU 201 or the like illustrated in FIG. 5.

The touch sensor 216 is not limited to the optical type, but may beformed by an electrostatic capacitance type touchscreen panel whichidentifies the contact position by detecting a change in electrostaticcapacitance. The touch sensor 216 may be a resistive film typetouchscreen panel which identifies the contact position by detecting avoltage change across two opposing resistive films. The touch sensor 216may be an electromagnetic induction type touchscreen panel whichidentifies the contact position by detecting an electromagneticinduction that is generated when the object contacts the touchscreenpanel (or display). Thus, the touch sensor 216 may use various detectionmeans. The touch sensor 216 may be of a type that does not require anelectronic pen to detect the presence or absence of the touch by the pentip. In this case, the user's fingertips, pen-shaped bars, or the likemay be used for the touch operations. The pen 2500 does not necessarilyneed to have the elongated pen shape.

<Functions of Display Device>

Next, functions of the display device 2 and the pen 2500 will bedescribed, with reference to FIG. 6A and FIG. 6B. FIG. 6A is afunctional block diagram illustrating an example of the functions of thedisplay device 2. The display device 2 includes a handwritten input part21, a display part 22, a handwritten input display control part 23, acandidate display timer control part 24, a handwritten input storagepart 25, a handwriting recognition control part 26, a handwritingrecognition dictionary part 27, a character string conversion controlpart 28, a character string conversion dictionary part 29, a predictiveconversion control part 30, a predictive conversion dictionary part 31,an operation command recognition control part 32, an operation commanddefinition part 33, a pen ID control data storage part 36, a handwrittensignature authentication control part 38, and a handwritten signaturedata storage part 39. Each function of the display device 2 is afunction or means implemented in one of the constituent elementsillustrated in FIG. 5 when the constituent elements perform an operationin response to the command from the CPU 201 according to the programloaded from the SSD 204 to the RAM 203 and executed by the CPU 201.

The handwritten input part 21 is implemented by the touch sensor 216 orthe like. The handwritten input part 21 receives the handwritten inputfrom the user, and the pen ID. The handwritten input part 21 converts auser's pen input d1 into pen operation data d2 (pen up, pen down, or pencoordinate data) with the pen ID, and transmits the pen operation datad2 to the handwritten input display control part 23. The pen coordinatedata is transmitted periodically as discrete values, and the coordinatesbetween the discrete values are calculated and complemented.

The display part 22 is implemented by the display 220 or the like, anddisplays a handwritten object or an operation menu. The display part 22converts rendered data d3 written into a video memory by the handwritteninput display control part 23, into data according to thecharacteristics of the display 220, and transmits the converted data tothe display 220.

The handwritten input display control part 23 performs an overallcontrol of the handwritten input and display. The handwritten inputdisplay control part 23 processes the pen operation data d2 from thehandwritten input part 21, and displays the processed pen operation datad2 by transmitting the same to the display part 22. Details of theprocesses performed on the pen operation data d2 and the display ofstrokes will be described later in conjunction with FIG. 33 through FIG.39.

The candidate display timer control part 24 includes a display controltimer for the selectable candidates. The candidate display timer controlpart 24 starts or stops the timer, and generates a timing for startingthe display of the selectable candidates, and a timing for deleting thedisplay. The selectable candidates include handwriting recognitioncharacter string candidates, language character string candidates,converted character string candidates, predictive converted characterstring candidates, and operation command candidates, that are selectablydisplayed on the operation guide which will be described later. Thecandidate display timer control part 24 receives a timer start requestd4 (or a timer stop request, as the case may be) from the handwritteninput display control part 23, and transmits a time out event d5 to thehandwritten input display control part 23.

The handwritten input storage part 25 includes a storage function thatstores user data (handwritten object/character string object). Thehandwritten input storage part 25 receives user data d6-1 from thehandwritten input display control part 23, and stores the user data d6-1in the handwritten input storage part 25. The handwritten input storagepart 25 receives an acquisition request d6-2 from the handwritten inputdisplay control part 23, and transmits user data d7 stored in thehandwritten input storage part 25. The handwritten input storage part 25transmits position information d36 of a decided object to the operationcommand recognition control part 32.

The handwriting recognition control part 26 includes an identificationengine for performing on-line handwriting recognition. Unlike thegeneral Optical Character Reader (OCR), characters (not only in Japanesecharacters but also characters of other languages, such as alphabets inthe case of the English language, for example), numbers, symbols (%, $,&, or the like), and graphics (lines, circles, triangles, or the like)are recognized in parallel with the user's pen operation. Variousalgorithms have been devised for the recognition technique, but adetailed description thereof will be omitted because this embodiment canutilize a conventional recognition technique that is appropriate.

The handwriting recognition control part 26 receives a pen operationdata d8-1 from the handwritten input display control part 23, performs ahandwriting recognition, and stores a handwriting recognition characterstring candidate. The handwriting recognition control part 26 stores alanguage character string candidate, converted from a handwritingrecognition character string candidate d12 using the handwritingrecognition dictionary part 27. In a case where an acquisition requestd8-2 is received separately from the handwritten input display controlpart 23, the handwriting recognition control part 26 transmits storedhandwriting recognition character string candidate and languagecharacter string candidate d9 to the handwritten input display controlpart 23.

The handwriting recognition dictionary part 27 includes dictionary datafor the language conversion of the handwriting recognition. Thehandwriting recognition dictionary part 27 receives a handwritingrecognition character string candidate d12 from the handwritingrecognition control part 26, converts the handwriting recognitioncharacter string candidate d12 into a language character stringcandidate d13 that is linguistically probable, and transmits theconverted language character string candidate d12 to the handwritingrecognition control part 26. For example, in the case of the Japaneselanguage, Hiragana characters are converted into Kanji characters orKatakana characters.

The character string conversion control part 28 controls the conversionof the converted character string candidate into a character string. Theconverted character string is likely generated to include thehandwriting recognition character string or the language characterstring. The character string conversion control part 28 receiveshandwriting recognition character string and language character stringcandidate d11 from the handwriting recognition control part 26, convertsthe handwriting recognition character string and language characterstring candidate d1 into a converted character string candidate usingthe character string conversion dictionary part 29, and stores theconverted character string candidate. In a case where an acquisitionrequest d14 is received separately from the handwritten input displaycontrol part 23, the character string conversion control part 28transmits a stored converted character string candidate d15 to thehandwritten input display control part 23.

The character string conversion dictionary part 29 includes dictionarydata for the character string conversion. The character stringconversion dictionary part 29 receives handwriting recognition characterstring and language character string candidate d17 from the characterstring conversion control part 28, and transmits a converted characterstring candidate d18 to the character string conversion control part 28.

The predictive conversion control part 30 receives handwritingrecognition character string and language character string candidate d10from the handwriting recognition control part 26. The predictiveconversion control part 30 receives a converted character stringcandidate d16 from the character string conversion control part 28. Thepredictive conversion control part 30 converts the handwritingrecognition character string, the language character string candidated10, and the converted character string candidate d16 into predictedcharacter string candidates using the predictive conversion dictionarypart 31, respectively. A predictive conversion character string islikely generated to include the handwriting recognition characterstring, the language character string or the converted character string.In a case where an acquisition request d19 is received separately fromthe handwritten input display control part 23, the predictive conversioncontrol part 30 transmits a predicted character string candidate d20 tothe handwritten input display control part 23.

The predictive conversion dictionary part 31 includes dictionary datafor the predictive conversion. The predictive conversion dictionary part31 receives the handwriting recognition character string and languagecharacter string candidate, and converted character string candidate d21from the predictive conversion control part 30, and transmits apredicted character string candidate d22 to the predictive conversioncontrol part 30.

The image transformation control part 42 receives, from the handwritteninput display control part 23, the handwriting recognition characterstring candidate converted by the handwriting recognition control part26 and the language character string candidate, the converted characterstring candidate converted by the character string conversion controlpart 28, and the predicted character string candidate converted by thepredictive conversion control part 30 (these character string candidateswill be designated by a reference numeral d56). The image transformationcontrol part 42 searches for character string candidates d58 receivedfrom the handwritten input display control part 23, starting from afront index of the image transformation dictionary part 43, and receivesa matching image candidate d59 from the image transformation dictionarypart 43. The image transformation control part 42 transmits a receivedimage candidate d57 to the handwritten input display control part 23.

The operation command recognition control part 32 receives handwritingrecognition character string and language character string candidate d30from the handwriting recognition control part 26. The operation commandrecognition control part 32 receives a converted character stringcandidate d28 from the character string conversion control part 28. Theoperation command recognition control part 32 receives a predictedcharacter string candidate d29 from the predictive conversion controlpart 30. The operation command recognition control part 32 transmits anoperation command conversion request d26 to the operation commanddefinition part 33 for the handwriting recognition character string, thelanguage character string candidate d30, the converted character stringcandidate d28, and the predicted character string candidate d29,respectively, and receives an operation command candidate d27 from theoperation command definition part 33. The operation command recognitioncontrol part 32 stores the operation command candidate d27.

In a case where the operation command conversion request d26 partiallymatches the operation command definition, the operation commanddefinition part 33 transmits the operation command candidate d27 to theoperation command recognition control part 32.

In addition, the operation command recognition control part 32 receivespen operation data d24-1 from the handwritten input display control part23. The operation command recognition control part 32 transmits aposition information acquisition request d23 of the decided object thatis input and decided in the past, to the handwritten input storage part25, and stores the decided object specified by the pen operation data,as a selected object (including position information). The operationcommand recognition control part 32 identifies the selected object thatsatisfies a predetermined criteria with the position of the penoperation data d24-1. In a case where a acquisition request d24-2 isreceived separately from the handwritten input display control part 23,the operation command recognition control part 32 transmits storedoperation command candidate and identified selected object d25 to thehandwritten input display control part 23.

The pen ID control data storage part (or storage means) 36 stores thepen ID control data. Before the handwritten input display control part23 transmits display data to the display part 22, the pen ID controldata storage part 36 transmits pen ID control data d41 to thehandwritten input display control part 23. The handwritten input displaycontrol part 23 renders display data under an operating condition storedin correspondence with the pen ID. Further, before the handwritingrecognition control part 26 executes the handwriting recognition, thepen ID control data storage part 36 transmits angular information d44 ofthe pen ID control data to the handwriting recognition control part 26,and the handwriting recognition control part 26 rotates the strokeaccording to the angular information stored in correspondence with thepen ID, to execute the handwriting recognition.

After the handwriting recognition control part 26 recognizes a straightline for setting the angular information when the user handwritescharacters or the like, the handwriting recognition control part 26transmits angular information d43 of the pen ID control data to the penID control data storage part 36, to store the angular information d43 incorrespondence with the pen ID. In addition, after the handwritten inputdisplay control part 23 executes the operation command for setting theangular information, the handwritten input display control part 23transmits pen ID control data d42 to the pen ID control data storagepart 36, to store the execution result of the operation command (theangular information set by the user) in correspondence with the pen ID.Thereafter, the stroke of the pen ID is rotated according to the setangular information, before executing the handwriting recognition.

The handwriting recognition control part 26 transmits stroke data d49,rotated clockwise according to the angular information of the pen IDcontrol data, to the handwritten signature authentication control part38. Hence, it becomes possible to authenticate the handwritten signatureregardless of the user's position (that is, the direction from which theuser makes the handwriting with respect to the display device 2).

The handwritten signature data storage part 39 stores the handwrittensignature data. When the handwritten signature data storage part 39receives a handwritten signature data acquisition request d45 from thehandwritten signature authentication control part 38, the handwrittensignature data storage part 39 transmits handwritten signature data d46to the handwritten signature authentication control part 38. A format ofthe handwritten signature data depends on the algorithm used by thehandwritten signature authentication control part 38 for the handwrittensignature authentication. The data of the handwritten signature datastorage part 39 will be described later in conjunction with FIG. 15.

When the stroke data d49 rotated clockwise is received from thehandwriting recognition control part 26, the handwritten signatureauthentication control part 38 transmits a handwritten signature dataacquisition request d45 to the handwritten signature data storage part39, and the handwritten signature data storage part 39 transmits ahandwritten signature data d46 to the handwritten signatureauthentication control part 38.

The handwritten signature authentication control part 38 authenticatesthe user based on the handwritten signature data. Various algorithmshave been devised for the user authentication based on the handwrittensignature data, and this embodiment may use a conventional techniquethat can recognize the handwritten signature data at a recognition ratethat does not adversely affect the user authentication from a practicalviewpoint. For example, a feature vector including coordinates formingthe handwritten signature data, a pen pressure, a stroke time, or thelike as elements thereof, may be generated, and the elements may beweighted, as appropriate. Then, a feature vector of the registeredhandwritten signature data, and the feature vector of a user name or thelike handwritten by the user at the time of signing in, may be comparedto determine a coincidence level of the compared feature vectors. It maybe determined that the user authentication is successful when thecoincidence level is greater than or equal to a threshold value, whileit may be determined that the user authentication is unsuccessful (thatis, failed) when the coincidence level is less than the threshold value.

The handwritten signature authentication control part 38 stores theauthentication result of the handwritten signature, which is a comparedresult of the stroke data d49 and the handwritten signature data d46. Ina case where an acquisition request d48 is received separately from thehandwritten input display control part 23, the handwritten signatureauthentication control part 38 transmits a stored authentication resultd47 of the handwritten signature to the handwritten input displaycontrol part 23. The authentication result of the handwritten signatureincludes information indicating whether or not the stroke data d49 andthe handwritten signature data d46 may be regarded as a match, and in amatching case, SignatureId, which will be described later, andcorresponding to the matching handwritten signature data d46.

In a case where the handwriting recognition result of the handwritingrecognition control part 26 is adapted to the operation commandinstructing execution of the handwritten signature registration, thehandwriting recognition control part 26 acquires data d52 input to ahandwritten signature registration form (a field to which thehandwritten signature data is input, as will be described later) fromthe handwritten input storage part 25. The handwriting recognitioncontrol part 26 transmits handwritten signature data d50 of the data d52to the handwritten signature authentication control part 38. Thehandwritten signature authentication control part 38 transmits thereceived handwritten signature data d50 to the handwritten signaturedata storage part 39, to register the same.

In a case where the handwriting recognition result of the handwritingrecognition control part 26 indicates a cancel instruction orregistration of the handwritten signature, the handwriting recognitioncontrol part 26 transmits a delete request d51 of the handwrittensignature registration form to the handwritten input storage part 25,and deletes the handwritten signature registration form from thehandwritten input storage part 25.

In a case where the handwriting recognition result of the handwritingrecognition control part 26 indicates an execute instruction to changethe user definition data, the handwriting recognition control part 26acquires data d53 input to a user definition data change form from thehandwritten input storage part 25. The handwriting recognition controlpart 26 transmits a changed value d54 of the data d53 to the operationcommand definition part 33, to change the user definition data. The userdefinition data will be described later in conjunction with FIG. 14.

In a case where the handwriting recognition result of the handwritingrecognition control part 26 indicates a cancel instruction orregistration of the user definition data change form, the handwritingrecognition control part 26 transmits a delete request d55 of the userdefinition data change form to the handwritten input storage part 25,and deletes the user definition data change form from the handwritteninput storage part 25.

FIG. 6B is a functional block diagram illustrating an example of thefunctions of the pen 2500. The pen 2500 includes a pen eventtransmitting part 41. The pen event transmitting part 41 transmits eventdata of pen up, pen down, and pen coordinates, together with the pen ID,to the display device 2.

<Defined Control Data>

Next, defined control data, used by the display device 2 for variousprocesses, will be described with reference to FIG. 7. FIG. 7illustrates an example of the defined control data. The exampleillustrated in FIG. 7 illustrates the control data for each controlitem.

A selectable candidate display timer 401 defines the time (one exampleof a first time) until the selectable candidate is displayed, so thatthe display device 2 does not display the selectable candidate while thehandwriting is being made. In FIG. 7 illustrates that the selectablecandidate is displayed unless a pen down occurs within a TimerValue=500[ms] from a pen up. The selectable candidate display timer 401 is storedby the candidate display timer control part 24. The selectable candidatedisplay timer 401 is used at the start of the selectable candidatedisplay timer in step S18-2 illustrated in FIG. 35 which will bedescribed later.

A selectable candidate delete timer 402 defines the time (one example ofa second time) until the displayed selectable candidate is deleted, sothat the selectable candidate is deleted if the user does not select theselectable candidate. In FIG. 7 illustrates that the selectablecandidate display data is deleted unless the selectable candidate isselected within a TimerValue=5000 [ms] from the display of theselectable candidate. The selectable candidate delete timer 402 isstored by the candidate display timer control part 24. The selectablecandidate delete timer 402 is used at the start of the selectablecandidate display delete timer in step S64 illustrated in FIG. 37 whichwill be described later.

A handwritten object rectangular region 403 defines a rectangular regionwhich may be regarded as being near the handwritten object. In theexample illustrated in FIG. 7, the handwritten object rectangular region403 expands the rectangular region of the handwritten object in thehorizontal direction by 50% of the estimated character size, and expandsthe rectangular region of the handwritten object in the verticaldirection by 80% of the estimated character size. In the exampleillustrated in FIG. 7, the estimated character size is indicated by apercentage (specified %). However, if the unit is specified as “mm” orthe like, the estimated character size may have a fixed length. Thehandwritten object rectangular region 403 is stored in the handwritteninput storage part 25. The estimated character size 405 is used in stepS10 illustrated in FIG. 34 which will be described later, to determinean overlapping state of the handwritten object rectangular region and astroke rectangular region.

An estimated writing direction/character size determination condition404 defines constants for determining the writing direction andcharacter size measuring direction. In the example illustrated in FIG.7, the estimated writing direction is determined to be “horizontalwriting”, and the estimated character size is determined to be thevertical distance, in a case where:

-   -   A difference MinTime between a time when the stroke is first        added and a time when the stroke is last added to the        handwritten object rectangular region is MinTime=1000 [ms] or        greater;    -   A difference MinDiff between the horizontal distance (width) and        the vertical distance (height) of the handwritten object        rectangular region is MinDiff=10 [mm] or greater; and    -   The horizontal distance is longer than the vertical distance.        In the case where the horizontal distance is shorter than the        vertical distance, the estimated writing direction is determined        to be “vertical writing” and the estimated character size is        determined to be the horizontal distance. In a case where the        above conditions are not satisfied, the estimated writing        direction is determined to be “horizontal writing”        (DefaultDir=“Horizontal”), and the estimated character size is        determined to be the longer distance between the horizontal        distance and the vertical distance. The estimated writing        direction/character size determination condition 404 is stored        in the handwritten input storage part 25. The estimated writing        direction/character size determination condition 404 is used for        acquiring the estimated writing direction in step S59        illustrated in FIG. 37, and for acquiring the character string        object font acquisition in step S81 illustrated in FIG. 39,        which will be described later.

An estimated character size 405 defines data for estimating the size ofthe characters or the like. In the example illustrated in FIG. 7, theestimated character size determined by the estimated writingdirection/character size determination condition 404 is compared to asmall character 405 a (hereinafter referred to as a minimum font size)of the estimated character size 405 and a large character 405 c(hereinafter referred to as a maximum font size). In a case where theestimated character size is smaller than the minimum font size, theestimated character size is determined to be the minimum font size. In acase where the estimated character size is larger than the maximum fontsize, the estimated character size is determined to be the maximum fontsize. Otherwise, the estimated character size is determined to be thecharacter size of a medium character 405 b. The estimated character size405 is stored in the handwritten input storage part 25. The estimatedcharacter size 405 is used for acquiring the character string objectfont in step S81 illustrated in FIG. 39, which will be described later.

More particularly, the handwritten input storage part 25 compares theestimated character size determined by the estimated writingdirection/character size determination condition 404 with FontSize ofthe estimated character size 405, and uses the font having the FontSizeclosest to the estimated character size. For example, the handwritteninput storage part 25 determines the estimated character size to be the“small character” when the estimated character size is 25 [mm] or less(FontSize of the small character), to be the “medium character” when theestimated character size is greater than 25 [mm] and 50 [mm] or less(FontSize of the medium character), and to be the “large character” whenthe estimated character size is greater than 100 mm (FontSize of thelarge character). The small character 405 a uses the 25 mm Ming font(FontStyle=“Ming”, FontSize=“25 mm”), the medium character 405 b usesthe 50 mm Ming font (FontStyle=“Ming”, FontSize=“50 mm”), and the largecharacter 405 c uses the 100 mm Gothic font (FontStyle=“Gothic”,FontSize=“100 mm”). The number of kinds of font sizes and styles can beincased, by increasing the number of kinds of the estimated charactersize 405.

A striding line determination condition 406 defines the data used fordetermining whether or not a plurality of objects are selected. Thehandwritten object is a single stroke. In the example illustrated inFIG. 7, it is determined that a plurality of objects are selected(selected objects are present), in a case where:

-   -   The length of the longer side of the handwritten object is 100        [mm] (MinLenLongSide=“100 mm”) or greater;    -   The length of the shorter side of the handwritten object is 50        [mm] (MaxLenShortSide=“50 mm”) or less; and    -   There is an object which overlaps the handwritten object with an        overlap ratio of 80 [%] (MinOverLapRate=“80%”) or higher along        the direction of the longer side and the direction of the        shorter side. The operation command recognition control part 32        stores the striding line determination condition 406. The        striding line determination condition 406 is used for        determining the striding line when determining the selected        object in step S50 illustrated in FIG. 36, which will be        described later.

An enclosure line determination condition 407 defines the data used fordetermining whether or not the object is an enclosure line. In theexample illustrated in FIG. 7, the operation command recognition controlpart 32 determines an object, that overlaps the handwritten object andis determined to have the overlap ratio of 100% (MinOverLapRate=“100%”)or higher along the direction of the longer side and the direction ofthe shorter side of the handwritten object, as the selected object. Theenclosure line determination condition 407 is stored in the operationcommand recognition control part 32. The enclosure line determinationcondition 407 is used for determining the enclosure line whendetermining the selected object in step S50 illustrated in FIG. 36,which will be described later.

The priority may be placed on the determination of either one of thestriding line determination condition 406 and the enclosure linedetermination condition 407. For example, in a case where the stridingline determination condition 406 is relaxed (set to a value so as tofacilitate selection of the striding line) and the enclosure linedetermination condition 407 is strict (set to a value so as to enableselection of only the enclosure line), the operation command recognitioncontrol part 32 may place the priority on the determination of theenclosure line determination condition 407.

<Example of Dictionary Data>

The dictionary data will be described with reference to FIG. 8 throughFIG. 10. FIG. 8 illustrates an example of the dictionary data of thehandwriting recognition dictionary part 27. FIG. 9 illustrates anexample of the dictionary data of the character string conversiondictionary part 29. FIG. 10 illustrates an example of the dictionarydata of the predictive conversion dictionary part 31. Each of thesedictionary data illustrated in FIG. 8 through FIG. 8 is used in stepsS33 through S42 illustrated in FIG. 36, which will be described later.

In this embodiment, the conversion result of the dictionary data of thehandwriting recognition dictionary part 27 illustrated in FIG. 8 will bereferred to as a language character string candidate, the conversionresult of the dictionary data of the character string conversiondictionary part 29 illustrated in FIG. 9 will be referred to as aconverted character string candidate, and the conversion result of thedictionary data of the predictive conversion dictionary part 31illustrated in FIG. 10 will be referred to as a predicted characterstring candidate. In addition, each dictionary data “before conversion”refers to the character string used for the search in the dictionarydata, each dictionary data “after conversion” refers to the characterstring after conversion and corresponding to the character string usedfor the search, and “probability” refers to the probability of theselection that will be made by the user. The probability may becalculated from the result of the user's selection of each characterstring made in the past. Accordingly, the probability may be calculatedfor each user. Various algorithms have been devised for the probabilitycalculation technique, but a detailed description thereof will beomitted because this embodiment can utilize a conventional probabilitycalculation technique that is appropriate. This embodiment may displaythe character string candidates in a descending order of the selectedprobability according to the estimated writing direction.

The dictionary data of the handwriting recognition dictionary part 27illustrated in FIG. 8 indicates that the handwritten Hiragana character“

” before the conversion and pronounced “gi” has a 0.55 probability ofbeing converted into a Kanji character “

” (which may mean “talk” or “consult” in English) after the conversionand also pronounced “gi” as indicated in the first line, and has a 0.45probability of being converted into another Kanji character “

” (which may mean “technical” in English) after the conversion and alsopronounced “gi” as indicated in the second line. The handwrittenHiragana characters “

” before the conversion and pronounced “gishi” has a 0.55 probability ofbeing converted into a character string of two Kanji characters “

” and also pronounced “gishi” after the conversion as indicated in thethird line, and has a 0.45 probability of being converted into anothercharacter string of two Kanji characters and also pronounced “gishi”after the conversion as indicated in the fourth line. The probabilitiesfor other handwritten Hiragana characters before the conversion, afterthe conversion, are indicated similarly in the fifth through eighthlines. Although FIG. 8 illustrates an example in which the handwrittencharacter string before the conversion are made up of Hiraganacharacters, characters other than the Hiragana characters may beregistered as the handwritten character string before the conversion.

The dictionary data of the character string conversion dictionary part29 illustrated in FIG. 9 indicates that the character string made up ofa Kanji character ‘

’ before the conversion and pronounced “gi” has a 0.95 probability ofbeing converted into a character string made up of a character string ofthree Kanji characters “

” after the conversion and pronounced “gijiroku” (which may mean“agenda” in English) as indicated in the first line, and anothercharacter string made up of another Kanji character “

” before the conversion and pronounced “gi” has a 0.85 probability ofbeing converted into another character string made up of three Kanjicharacters “

” after the conversion and pronounced “giryoushi” (which may mean“qualification trial” in English) as indicated in the second line. Theprobabilities for other character strings before the conversion, afterthe conversion, are indicated similarly in the third through tenthlines.

The dictionary data of the predictive conversion dictionary part 31illustrated in FIG. 10 indicates that the character string made up ofthree Kanji characters “

” before the conversion and pronounced “gijiroku” (which may mean“agenda” in English”) has a 0.65 probability of being converted into acharacter string made up of seven Kanji and Hiragana characters “

” after the conversion and pronounced “gijiroku no soufusaki” (which maymean “sending destination of agenda” in English) as indicated in thefirst line, and another character string made up of three Kanjicharacters “

” before the conversion and pronounced “giryoushi” (which may mean“qualification trial” in English) has a 0.75 probability of beingconverted into a character string made up of six Kanji and Hiraganacharacters “

” after the conversion and pronounced “giryoushi wo kessai” (which maymean “qualification trial approval” in English) as indicated in thesecond line. The probabilities for other character strings before theconversion, after the conversion, are indicated similarly in the thirdthrough eighth lines. Although FIG. 10 illustrates an example in whichall of the character strings before the conversion are made up of Kanjicharacters, characters other than Kanji characters may be registered asthe character string before the conversion.

The dictionary data requires no language dependency, and any characterstring may be registered before and after the conversion.

FIG. 11 illustrates an example of image dictionary data included in theimage transformation dictionary part 43. The image obtained by making asearch in the image transformation dictionary part 43 will be referredto as an image candidate. The image dictionary data includes an indexcolumn, and an image column. The index column indicates the name of theimage. The name in the index column is used for the comparison todetermine whether or not a forward match with the handwritingrecognition character string candidates, the language character stringcandidates, the converted character string candidates, or the predictedcharacter string candidates occurs. Because the handwriting recognitioncharacter string candidates, the language character string candidates,the converted character string candidates, or the predicted stringcandidates are input in stages according to the user's handwriting, thematch is determined when the search in the index column reveals aforward match. For example, when the user inputs a character string madeup of two Katakana characters “

” pronounced “asu”, the forward match occurs for a character string madeup of seven Katakana and Kanji characters “

” pronounced “asukorubinsan” (“ascorbic acid” in English) and acharacter string made up of seven Katakana and Kanji characters “

” pronounced “asuparaginsan” (“aspartic acid” in English).

The image in the image column may be any visually understandable imageincluding figures such as diagrams or the like, illustrations such asbit maps or the like, photographs, 3-Dimensional Computer Graphics(3DCGs), or the like. The illustrations depict or decorate stories,novels, poems, or the like by the figures, to assist the characterinformation of sciences, news reports, or the like. The illustrationsare graphic or pictorial visual representations focusing on the subjectmatter rather than the format. The illustrations may be pictures,pictograms, or the like that are created according to the needs. The3DCGs represent a 3-dimensional object using polygons, point group data,or the like, or are data of the 3-dimensional object. The 3-dimensionalstructure enables a viewer to view the shape from directions covering360 degrees.

In addition, a file format of the image may be any one of JointPhotographic Experts Group (JPEG), Graphics Interchange Format (GIF),Portable Network Graphics (PNG), Tagged Image File Format (TIFF), BitmapImage File Format (BMP), or the like. In the case of the 3DCG, a fileformat suitable for the 3DCG may be used. Further, the display device 2may refer to the internal image dictionary data, or may retrieve andrefer to the image searched on the Internet by a search engine.

<Operation Command Definition Data Stored in Operation CommandDefinition Part>

Next, operation command definition data used by the operation commandrecognition control part 32 will be described, with reference to FIG.12A, FIG. 12B, and FIG. 13. FIG. 12A and FIG. 12B illustrate an exampleof the operation command definition data and system definition datastored in the operation command definition part 33.

FIG. 12A illustrates an example of the operation command definitiondata. The operation command definition data illustrated in FIG. 12A isan example of the operation command definition data for a case wherethere is no selected object selected by the handwritten object, and alloperation commands that operate the display device 2 are targets. Theoperation command illustrated in FIG. 12A includes an operation commandname (Name), a character string (String) that partially matches thecharacter string candidate, and an operation command character string(Command) to be executed. “%˜%” in the operation command characterstring is a variable, and corresponds to the system definition data asillustrated in FIG. 12B. In other words, “%˜%” is replaced by the systemdefinition data illustrated in FIG. 12B.

First, in operation command definition data 701 illustrated in FIG. 12A,the operation command name is a character string made up of fourteenKanji, Katakana, and Hiragana characters “

” pronounced “gijiroku tenpureito wo yomikomu” (“load agenda template”in English), the character string that partially matches the characterstring candidate is made up of three Kanji characters “

” pronounced “gijiroku” (“agenda” in English) or a character string madeup of six Katakana characters “

” pronounced “tenpureito” (“template” in English), and the operationcommand character string to be executed is “ReadFilehttps://%username%:%password%@server.com/template/minutes.pdf”. In thisexample, the system definition data “%˜%” is included in the operationcommand character string to be executed, and “%username%” and“%password%” are replaced by system definition data 704 and 705,respectively. Accordingly, the final operation command character stringis “ReadFile https://taro.tokkyo:x2PDHTyS@server.com/template/minutes.pdf”, indicating a read file(ReadFile) “https://taro.tokkyo:x2PDHTyS@server.com/template/minutes.pdf”.

In operation command definition data 702, the operation command name isa character string made up of thirteen Kanji, Katakana, and Hiraganacharacters “

” pronounced “gijiroku foruda ni hozonsuru” (“store agenda folder” inEnglish), the character string that partially matches the characterstring candidate is three Kanji characters “

” pronounced “gijiroku” (“agenda” in English) or two Kanji characters “

” pronounced “hozon” (“store” in English), and the operation commandcharacter string to be executed is “WriteFilehttps:/%username%:%password%@server.com/minutes/%machinename%_%yyyyy-mm-dd%.pdf”. Similar to the operation command definitiondata 701, “%username%”, “%password%”, and “%machinename%” in theoperation command character string are replaced by the system definitiondata 704, 705, and 706 illustrated in FIG. 12B, respectively.“%yyyy-mm-dd%” is replaced by the current year, month, and date. Forexample, if the current date is Sep. 26, 2018, “%yyyy-mm-dd%” isreplaced by “2018-09-26”. The final operation command is “WriteFilehttps://taro.tokkyo:x2PDHTyS@server.com/minutes/%My-Machine_2018-09-26.pdf”, indicatingstoring (writing) “gijiroku” (“agenda” in English) in a write file(WriteFile) “https://taro.tokkyo: x2PDHTyS

server.com/% minutes/%My-Machine_2018-09-26.pdf”.

In operation command definition data 703, the operation command name isa character string made up of four Kanji and Hiragana characters “

” pronounced “insatsu suru” (“print” in English)”, the character stringthat partially matches the character string candidate is made up of twoKanji characters “

” pronounced “insatsu (“print” in English)” or a character string madeup of four Katakana characters “

” pronounced “purinto” (“print” in English)”, and the operation commandcharacter string to be executed is “PrintFile https:/%username%:%password%

server.com/print/%machiname%-%yyyy-mm-dd%.pdf”. When the operationcommand character strings are replaced as in the operation commanddefinition data 702, the final operation command is “PrintFilehttps://taro.tokkyo: x2PDHTyS

server.com/print/%My-Machine_2018-09-26.pdf”, indicating that the file(PrintFile) “https://taro.tokkyo: x2PDHTyS

server.com/print/%My-Machine_2018-09-26.pdf” is printed (PrintFile),that is, the file is transmitted to a server. The printer prints thecontents of the file on paper when the user causes the printer tocommunicate with the server and specifies the file.

As described above, because the operation command definition data 701through 703 can be identified from the character string candidates, theoperation command can be displayed when the user handwrites theoperation command. Further, in a case where the user authentication issuccessful, “%username%”, “%password%”, or the like of the operationcommand definition data are replaced by the user information, and thus,the input and output of the file, in correspondence with the user,becomes possible.

In a case where the user authentication is not performed (including acase where the user authentication fails but the user is able to use thedisplay device 2), the display device 2 is replaced by “%username%”,“%password%”, or the like of the display device 2 that is preset.Accordingly, even without the user authentication, the input and outputof the file in correspondence with the display device 2 becomespossible.

Operation command definition data 709, 710, and 711 are operationcommands for changing the pen state. The pen state may also be referredto as the pen type. In the operation command definition data 709, 710,and 711, the names of the operation commands are a character string madeup of three Kanji and Katakana characters “

” pronounced “hoso pen” (“thin pen” in English), a character string madeup of three Kanji and Katakana characters “

” pronounced “futo pen” (“thick pen” in English), and a character stringmade up of four Katakana characters “

” pronounced “markar” (“marker” in English), respectively, the characterstrings that partially match the character string candidates are oneKanji character “

” pronounced “hoso” (“thin” or “fine” in English) or a character stringmade up of two Katakana characters “

” pronounced “pen” (“pen” in English), one Kanji character pronounced“futo” (“thick” of “bold” in English) or a character string made up oftwo Katakana characters “

” pronounced “pen” (“pen” in English), and a character string made up offour Katakana characters “

” pronounced “markar” (“marker” in English), respectively, and theoperation command character strings are “ChangePen fine”, “ChangePenbold”, and “ChangePen marking”, respectively. When these operationcommands are executed, the pen state is stored in the pen ID controldata storage part 36, and thus, the user can handwrite the strokes inthe set pen state.

Operation command definition data 712 is an operation command foraligning the orientation of the text data in a predetermined direction.In the operation command definition data 712, the operation command nameis a character string made up of ten Katakana, Kanji, and Hiraganacharacters “

” pronounced “tekisuto houkou wo soroeru” (“align text direction” inEnglish), the character string that partially matches the characterstring candidate is made up of four Katakana characters “

” pronounced “tekisuto” (“text” in English), a character string made upof two Kanji and Hiragana characters “

” pronounced “muki” (“orientation” in English), or a character stringmade up of two Kanji characters “

” pronounced “houkou” (“direction” in English), and the operationcommand character string is “AlignText Direction”. The text data writtenby the user from a direction other than the top to bottom direction withrespect to the touch sensor 216 will have a random orientation, therebymaking it is difficult to read all of the text data from one direction.When the user executes the operation command definition data 712, thedisplay device 2 aligns the recognized handwritten character strings inthe same direction (for example, in a direction aligned with respect tothe top to bottom direction). In this case, aligning the text datarefers to rotating only angular information of the text data.

In operation command definition data 713, the operation command name isa character string made up of ten Kanji, Hiragana, and Katakanacharacters pronounced “tegaki sain touroku suru” (“register handwrittensignature” in English), the character strings that partially match thecharacter string candidates are made up of three Katakana characters “

” pronounced “sain” (“signature” in English) and two Kanji characters “

” pronounced “touroku” (“register” in English), and the operationcommand character string is “RegisterSignature”. When the“RegisterSignature” command is executed, a handwritten signatureregistration form is added to the handwritten input storage part 25, andthe handwritten signature registration form is displayed on theoperation screen 101. An example of the handwritten signatureregistration form will be described later, by referring to FIG. 30Athrough FIG. 30C, for example.

In operation command definition data 714, the operation command name isa character string made up of ten Kanji, Hiragana, and Katakanacharacters “

” pronounced “tegaki sain in on” (“handwritten sign in on” iPnPgRsh),the character string that partially matches the character stringcandidate is “%signature%”, and the operation command is “Signin”. Thecharacter string “%signature%” is a reserved word of the systemdefinition data, and represents the fact that the registered handwrittensignature data and the stroke data of the user name or the like meet thecriteria for a match. In other words, when the registered handwrittensignature data and the stroke data meet the criteria for the match, theoperation command 512 based on the operation command definition data 714is displayed in the operation guide 500, as illustrated in FIG. 31 whichwill be described later.

When the “Signin” command is executed, “AccountId” of the user having“SignatureId” of the matching handwritten signature data, is stored inthe pen ID control data of the pen 2500 used to handwrite the user nameor the like by the stroke data. Hence, the pen ID and the “AccountId”are made to correspond to each other, and the display device 2 can usethe user definition data identified by the “AccountId”, as illustratedin FIG. 17A which will be described later.

In operation command definition data 715, the operation command name isa character string made up of eleven Kanji, Hiragana, and Katakanacharacters “

” pronounced “tegaki sain auto suru” (“handwritten sign out” inEnglish), the character string that partially matches the characterstring candidate is made up of three Katakana characters “

” pronounced sain“si” in English) or three Katakana characters “

” pronounced “auto” (“out” in English), and the operation command is“Signout”. When the “Signout” command is executed, the “AccountId” isdeleted from the pen ID control data of the pen 2500 that is used forthe handwritten sign out. Hence, the correspondence between the pen IDand the “AccoundId” is deleted, thereby making it possible for anarbitrary user to use the pen 2500.

In operation command definition data 716, the operation command name isa character string made up of six Kanji and Hiragana characters “

” pronounced “settei henkou suru” (“change setting” in English), thecharacter string that partially matches the character “

” string candidate is made up of two Kanji characters pronounced“settei” (“set” in English) or two Kanji characters “

” pronounced “henkou” (“change” in English), and the operation commandis “ConfigSettings”. When the “ConfigSettings” command is executed, auser definition data change form is added to the handwritten inputstorage part 25, and the user definition data change form is displayedon the operation screen 101. The user definition data change form willbe described later, by referring to FIG. 32A and FIG. 32B, for example.

Next, the operation command definition data when the handwritten objectis present, that is, the operation command definition data for anediting system and a decorating system, will be described. FIG. 13illustrates an example of the operation command definition data when aselected object selected by the handwritten object is present. Theoperation command definition data illustrated in FIG. 13 includes anoperation command name (Name), a group name (Group) of the operationcommand candidates, and an operation command character string (Command)to be executed.

Operation command definition data 707 defines the operation commands forthe editing system (Group=“Edit”), and is an example of the definitiondata of the operation command names for the editing system, including acharacter string made up of two Kanji characters “

” pronounced “shoukvo” (“delete” in English), a character string made upof two Kanji characters “

” pronounced “idou” (“move” in English), a character string made up oftwo Kanji characters “

” pronounced “kaiten” (“rotate” in English), and a character string madeup of two Kanji characters “

” pronounced “sentaku” (“select” in English). In other words, theseoperation commands are displayed with respect to the selected objects,so that the user can select a desired operation command.

Operation command definition data 708 defines the operation commands forthe decorating system (Group=“Decorate”), and is an example of thedefinition data for the operation command names for the decoratingsystem, including a character string made up of two Kanji and Hiraganacharacters “

” pronounced “futoku” (“thick” in English), a character string made upof two Kanji and Hiragana characters “

” pronounced “hosoku” (“thin” in English), a character string made up ofthree Kanji and Hiragana characters “

” pronounced “ookiku” (“large” in English), a character string made upof three Kanji and Hiragana characters “

” pronounced “chiisaku” (“small” in English), and a character stringmade up of two Kanji characters “

” pronounced “kasen” (“underline” in English). These operation commandsare displayed with respect to the selected objects, so that the user canselect a desired operation command. Other operation command, such asoperation commands related to color, may also be displayed.

Accordingly, the operation command definition data 707 and 708 areidentified when the user specifies the decided object by the handwrittenobject, so that the user can cause the operation command to be displayedby making the handwriting.

<User Definition Data>

Next, user definition data will be described with reference to FIG. 14.FIG. 14 illustrates an example of the user definition data stored by theoperation command definition part 33. The user definition dataillustrated in FIG. 14 is an example of the definition data of a singleuser. “AccountId” of user definition data 717 is user identificationinformation (or numbering) automatically assigned to each user,“AccountUsername” and “AccountPassword” are a user name and a password,respectively, “SignatureId” is identification information (or numbering)automatically assigned to the handwritten signature data at the time ofregistering the handwritten signature data, and “username”, “password”,and “machinename” are character strings set in the operation commanddefinition data 701, 702, and 703 in place of the system definition data704, 705, and 706, respectively. Hence, it is possible to execute anoperation command using the user definition data.

In a case where the user signs in by handwriting the user name or thelike, the character string of the user definition data including the“AccountId” in correspondence with the pen ID of the pen 2500 used bythe user, is used when executing the operation command by utilizing thecorrespondence between the pen ID and the “AccountId” with respect tothe pen ID control data, as illustrated in FIG. 17A. After the usersigns out, the character string for the system definition data is usedwhen executing the operation command, even if the pen 2500 that is usedby the user to sign in is used.

User definition data 718 is used in the user definition data changeform. “Name” is the item name of “AccountUsername”, “AccountPassword”,“username”, “password”, or “machinename” of the user definition data717, and “Data” is the changed value of “AccountUsername”,“AccountPassword”, “username”, “password”, or “machinename”. In thisexample, the data for a character string made up of two Kanji characters“

” pronounced “namae” (“name” in English) is “% AccountName %”. The datafor a character string made up of five Katakana characters “

” pronounced “pasuwado” (“password” in English) is “%AccountPassword%”.The data for a character string made up of ten Katakana and Kanjicharacters “

” pronounced “foruda yuza mei” (“folder user name” in English) is“%username%”. The data for a character string made up of ten Katakanacharacters “

” pronounced “foruda pasuwado” (“folder password in English) is“%password%”. The data for a character string made up of ten Katakanaand Kanji characters “

” pronounced “foruda fairu mei” (“folder password” data is“%machinename%”. Each item corresponds to each item of user definitiondata 717. These items input to the user definition data change form arereflected in the user definition data 717.

<Handwritten Signature Data>

Next, the handwritten signature data will be described with reference toFIG. 15. FIG. 15 illustrates an example of the handwritten signaturedata stored in the handwritten signature data storage part 39. Thehandwritten signature data includes “Data” representing the handwrittensignature corresponding to “SignatureId”. “SignatureId” isidentification information (or numbering) automatically assigned at thetime of registering the handwritten signature data, and “Data” is datacalculated from the stroke data received from the handwritten signatureauthentication control part 38, according to a handwritten signatureauthentication algorithm of the handwritten signature authenticationcontrol part 38.

<Handwritten Input Storage Data Stored in Handwritten Input StoragePart>

Next, handwritten input storage data will be described, with referenceto FIG. 16. FIG. 16 illustrates an example of the handwritten inputstorage data stored in the handwritten input storage part 25. One linein FIG. 16 represents one stroke. One handwritten input storage dataincludes each of items “DataId”, “Type”, “PenId”, “Color”, “Width”,“Pattern”, “Angle”, “AccountId”, “StartPoint”, “StartTime”, “EndPoint”,“EndTime”, “Point”, and “Pressure”.

“DataId” represents identification information of the stroke. “Type”represents a type of the stroke. The types include the stroke (Stroke),the group (Group), and the text (Text). The type of handwritten inputstorage data 801 and 802 is “Stroke”, and the type of handwritten inputstorage data 803 is “Group”. “Group” encompasses grouping the strokewith other strokes, and the handwritten input storage data having thetype “Group” specifies a target stroke to be grouped by “DataId”.“PenId”, “Color”, “Width”, “Pattern”, “Angle”, and “AccountId” aretranscribed from the pen ID control data described below. “StartPoint”represents start point coordinates of the stroke, “StartTime” representsa start time of the stroke, “EndPoint” represents end point coordinatesof the stroke, and “EndTime” represents an end time of the stroke.“Point” represents a sequence of coordinate points from the start pointto the end point, and “Pressure” represents the pen pressure from thestart point to the end point. As indicated by “Angle”, handwritten inputstorage data 804 and 805 indicate the handwriting recognition afterbeing rotated clockwise by 180 degrees and 270 degrees, respectively.The handwritten input storage data 802 and 805 indicate that thehandwritten input storage data 802 and 805 are input by the user havingAccountId=1 for the user definition data.

<Pen ID Control Data Stored in Pen ID Control Data Storage>

Next, the pen ID control data will be described, with reference to FIG.17A and FIG. 17B. FIG. 17A is a diagram illustrating the pen ID controldata stored in the pen ID control data storage part 36. One line in FIG.17A represents the pen ID control data of one pen. FIG. 17B is a diagramfor explaining angular information when the user makes the handwritingwith respect to the display device 2. The angular information may be theangle of the direction in which the user is present, the angle of thedirection in which the pen is used, or the angle of the rotation of thecharacters handwritten by the user. When a predetermined direction (forexample, the top to bottom direction) of the display device 2 isregarded as being the direction with the angle of 0 degrees (reference),the angular information of each of the users is 45 degrees, 90 degrees,135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees,counterclockwise.

In the case where the display device 2 is placed flat, the angularinformation of the user represents the position of the user with respectto the display device 2. In other words, the angular information of theuser may be regarded as being the information related to the position ofthe user. The display device 2 can identify the direction in which theuser is present when viewed from the display device 2. The angularinformation may be replaced by other information, such as informationexpressing 0 degrees as a 6 o'clock direction, 45 degrees as in-betweena 4 o'clock direction and a 5 o'clock direction, 90 degrees as a 3o'clock direction, 135 degrees as in-between a 1 o'clock direction and a2 o'clock direction, 180 degrees as a 12 o'clock direction, 225 degreesas in-between a 10 o'clock direction and a 11 o'clock direction, 270degrees as a 9 o'clock direction, and 315 degrees as in-between a 7o'clock direction and a 8 o'clock direction, based on the directionviewed from the display device 2.

The angular information is not automatically determined by the user'sposition, and each user inputs (specifies) the angular information. Aresolution of the angular information that can be specified in thisexample (45 degrees in FIG. 17A and FIG. 17B) is merely one example, andthe resolution may be set to a smaller angle in a range of 5 degrees to30 degrees, for example. However, if the characters are rotated by anangle of approximately 45 degrees, the user is likely able to read andvisually recognize the characters.

The pen ID control data includes “PenId”, “Color”, “Width”, “Pattern”,“Angle”, and “AccountId”. “PenId” is identification information storedinside the pen. “Color” is the color of the stroke set in this pen, andcan be changed arbitrarily by the user. “Width” is the thickness (orwidth) of the stroke set in this pen, and can be changed arbitrarily bythe user. “Pattern” is the line type of the stroke set in this pen, andcan be changed arbitrarily by the user. “Angle” is the angularinformation of the stroke set in this pen, and can be changedarbitrarily by the user. In the example illustrated in FIG. 17A, theangular information of each pen is 0 degrees, 90 degrees, 180 degrees,and 270 degrees, counterclockwise. AccountId is the user's identity.Because “AccountId” is made to correspond to the pen ID, it is possibleto identify the “AccountId” corresponding to the pen ID of the pen 2500that is used by the user, and execute the operation command using theuser definition data.

Pen ID control data 901 is the control data in which the pen ID is 1, acolor that is black (Black), a thickness that is 1 pixel (1 px), apattern that is solid (Solid), angular information that is 0 degrees,and AccountId=1. The user with AccountId=1 is the user with the userdefinition data 717 illustrated in FIG. 14, and indicates that this usersigned in by handwriting the user name or the like using the pen withthe pen ID of 1. The pen ID control data having no “AccountId” indicatesa signed out state (not made to correspond to the user).

Similarly, pen ID control data 902 is the control data in which the penID is 2, a color that is black, a thickness that is 1 pixel, a patternthat is solid, angular information that is 90 degrees, and includes noAccountId. Pen ID control data 903 is the control data in which the penID is 3, a color that is black, a thickness that is 10 pixels, a patternthat is solid, angular information that is 180 degrees, and includes noAccountId. Pen ID control data 904 is the control data in which the penID is 4, a color that is black, a thickness that is 10 pixels, a patternthat is a halftone dot pattern, angular information that is 270 degrees,and includes no AccountId.

The control data is used in step S5 illustrated in FIG. 33 for acquiringthe pen ID control data illustrated in FIG. 33, in step S20 illustratedin FIG. 35 for storing the angular information of the pen ID controldata, in step S21 illustrated in FIG. 35 for acquiring the angularinformation of the pen ID control data, in step S60 illustrated in FIG.37 for acquiring the pen ID control data, and in step S88 illustrated inFIG. 39 for storing the angular information of the pen ID control data.

<Example of Selectable Candidates>

First, a selectable candidate displaying only characters will bedescribed with reference to FIG. 18. FIG. 18 illustrates an example of aselectable candidate 530 displayed by an operation guide and anoperation guide. When the user handwrites the handwritten object 504 anda time out of the selectable candidate display timer occurs, theoperation guide 500 is displayed. The operation guide 500 includes anoperation header 520, an operation command candidate 510, a handwritingrecognition character string candidate 506, a converted character stringcandidate 507, a character string/predictive conversion candidate 508,and a handwritten object rectangular area display 503. The selectablecandidate 530 includes an operation command candidate 510, a handwritingrecognition character string candidate 506, a converted character stringcandidate 507, and a character string/predictive conversion candidate508. A language character string candidate is not displayed in thisexample, however, the language character string candidate may bedisplayed, as appropriate. The selectable candidate 530, excluding theoperation command candidate 510, will be referred to as a characterstring candidate 539.

The operation header 520 includes buttons 501, 509, 502, and 505. Thebutton 501 accepts a switching operation between the predictiveconversion and the Kana conversion. In the example illustrated in FIG.18, when the user presses the button 509 indicating a character stringmade up of two Kanji characters “

” pronounced “yosoku” (“predictive” in English) to select the predictiveconversion, the handwritten input part 21 accepts the selectedpredictive conversion and notifies the same to the handwritten inputdisplay control part 23, and the display part 22 changes the display ofthe button 509 to indicate a character string made up of two Hiraganacharacters “

” pronounced “kana” to enable selection of the Kana conversion. Afterthis change, the character string candidate 539 arranges the candidatesin a descending probability order of the Kana conversion which convertsthe Hiragana characters into the Kanji and/or Katakana characters.

The button 502 accepts a page operation on the candidate display. In theexample illustrated in FIG. 18, there are three candidate display pages,and the first page is currently displayed. The button 505 acceptsdeletion of the operation guide 500. When the user presses the button505, the handwritten input part 21 accepts the deletion and notifies thesame to the handwritten input display control part 23, and the displaypart 22 deletes the display other than the handwritten object. Thebutton 509 accepts collective display deletion. When the user pressesthe button 509, the handwritten input part 21 accepts the collectivedisplay deletion and notifies the same to the handwritten input displaycontrol part 23, and the display part 22 deletes all of the displayillustrated in FIG. 18, including the handwritten object. Accordingly,the user can redo the handwriting from the start.

The handwritten object 504 in this example is a Hiragana character “

” pronounced “gi”. The handwritten object rectangular area display 503,surrounding the handwritten object 504, is displayed. The displayprocedure may be performed in the sequence described later inconjunction with FIG. 33 through FIG. 39. In the example illustrated inFIG. 18, the handwritten object rectangular area display 503 isdisplayed as a rectangular frame indicated by dots.

The handwriting recognition character string candidate 506, theconverted character string candidate 507, and the characterstring/predictive conversion candidate 508 respectively includecharacter string candidates arranged in the descending probabilityorder. The Hiragana character “

” pronounced “gi” of the handwriting recognition character stringcandidate 506 is the candidate of the recognition result. In thisexample, the display device 2 correctly recognizes the Hiraganacharacter “

” pronounced “gi”.

The converted character string candidate 507 is the converted characterstring candidate converted from the language character string candidate.In this example, the converted character string candidate 507 displaysthe upper character string made up of three Kanji characters “

” pronounced “gijiroku” (which may mean “agenda” in English), and thelower c e string made up of three Kanji characters “

” pronounced “giryoushi” (which may mean “qualification trial” inEnglish), which is an abbreviation for a character string made up of sixKanji characters “

” pronounced “gijutsu ryousan shisaku” (which may mean “technical massproduction trial” in English). The character string/predictiveconversion candidate 508 is the predicted character string candidateconverted from the language character string candidate or the convertedcharacter string candidate. In this example, the characterstring/predictive conversion candidate 508 displays the upper characterstring made up of six Kanji and Hiragana characters “

” pronounced “giryoushi wo kessai” (which may mean “qualification trialapproval” in English), and the lower character string made up of sevenKanji and Hiragana characters “

” after the conversion and pronounced “gijiroku no soufusaki” (which maymean “sending destination of agenda” in English).

The operation command candidate 510 is the operation command candidateselected based on the operation command definition data 701 through 703,and 709 through 716 illustrated in FIG. 12A. In the example illustratedin FIG. 18, a bullet character “>>” 511 indicates the operation commandcandidate. In FIG. 18, there is no decided object that can be selectedby the handwritten object 504 that is a Hiragana character pronounced“gi”, and because the character string “

” candidate (upper character string) made up of three Kanji characters “

” pronounced “gijiroku” (which may mean “agenda” in English) displayedin the converted character string candidate 507, which is the characterstring candidate of the handwritten object 504, partially matches theoperation command definition data 701 and 702 illustrated in FIG. 12A,the character string candidate (upper character string) made up of threeKanji characters “

” pronounced “gijiroku” (which may mean “agenda” in English), displayedin the converted character string candidate 507, is displayed as theoperation command candidate 510 of the operation command.

The operation command candidate 510 includes an upper candidate (uppercharacter string) made up of fourteen Kanji, Katakana, and Hiraganacharacters “

” pronounced “gijiroku tenpureto wo yomikomu” (“load agenda template” inEnglish), and a lower candidate (lower character string) made up ofthirteen Kanji, Katakana, and Hiragana characters “

” Pronounced “gijiroku foruda ni hozonsuru” (“store agenda folder” inEnglish). When the user selects the upper candidate following the upperbullet character “>>” 511 displayed in the operation command candidate510, the operation command defined by the operation command definitiondata 701 is executed. When the user selects the lower candidatefollowing the lower bullet character “>>” 511 displayed in the operationcommand candidate 510, the operation command defined by the operationcommand definition data 702 is executed. Because the operation commandcandidate is displayed when the operation command definition dataincluding the converted character string is found, the operation commandcandidate is not always displayed.

As illustrated in FIG. 18, the character string candidates and theoperation command candidates are displayed together at the same time,and thus, the user can arbitrarily select each of the character stringcandidate and the operation command candidate intended by the user.

Next, the operation guide 500 which displays only image candidates willbe described, with reference to FIG. 19 through FIG. 21. FIG. 19 throughFIG. 21 illustrate transition states in which image candidates 531 aregradually narrowed down according to the user's handwriting. As will bedescribed below, the handwriting recognition control part 26successively converts the handwritten data from the beginning, toconvert the handwritten data into characters. The image transformationcontrol part 42 performs a search in the image transformation dictionarypart 43 by the forward match, while increasing the number of convertedcharacters every time the character is converted. The display part 22displays a number of image candidates, which decreases as the number ofcharacters used for the search increases.

FIG. 19 illustrates a case where the user handwrites one Katakanacharacter “

” pronounced “

” as the handwritten object 504. When the Katakana character of thehandwritten object 504 is recognized, and the Katakana character “

” pronounced “

” is obtained as the handwriting recognition character string, thehandwritten input display control part 23 transmits the Katakanacharacter “

” pronounced “

” to the image transformation control part 42. The image transformationcontrol part 42 searches the image transformation dictionary part 43illustrated in FIG. 11, and transmits image candidates corresponding toan index column having a name meeting the criteria for the match withthe Katakana character “

” pronounced “

” when the forward match is performed, to the handwritten input displaycontrol part 23. As a result, in the example illustrated in FIG. 19,five image candidates 531 having names beginning with the Katakanacharacter “

” pronounced “

” are displayed in the operation guide 500.

Next, FIG. 20 illustrates the image candidates 531 displayed when theuser handwrites the Katakana character “

” pronounced “

”, and thereafter handwrites another Katakana character “

” pronounced “se”, without selecting the image candidate 531.Accordingly, because the user successively handwrites two Katakanacharacters “

” pronounced “ase”, the handwritten input display control part 23transmits the two Katakana characters “

” pronounced “ase” to the image transformation control part 42. Theimage transformation control part 42 searches the image transformationdictionary part 43 illustrated in FIG. 11, and transmits imagecandidates corresponding to an index column having a name meeting thecriteria for the match with the two Katakana characters “

” pronounced “ase” when the forward match is performed, to thehandwritten input display control part 23. As a result, in the exampleillustrated in FIG. 20, two image candidates 531 having names beginningwith the two Katakana characters “

” pronounced “ase” are displayed in the operation guide 500.

As described above, the image transformation control part searches theimage transformation dictionary by performing the forward match, whileincreasing the number of converted characters every time the characteris converted.

Next, FIG. 21 illustrates the image candidate 531 displayed when theuser handwrites the character string made up of eight Katakanacharacters “

” (all characters indicating “acetaldehyde” in Japanese) pronounced“asetoarudehido” (“ascorbic acid” in English). The handwritten inputdisplay control part 23 transmits the character string made up of eightKatakana characters “

” pronounced “asetoarudehido” to the image transformation control part42. The image transformation control part 42 searches the imagetransformation dictionary part 43 illustrated in FIG. 11, and transmitsimage candidate corresponding to an index column having a name meetingthe criteria for the match with the character string made up of eightKatakana characters “

” pronounced “asetoarudehido” when the forward match is performed, tothe handwritten input display control part 23. As a result, in theexample illustrated in FIG. 21, one image candidate 531 having the namethat is the character string made up of eight Katakana characters “

” pronounced “asetoarudehido” is displayed in the operation guide 500.

As illustrated in FIG. 19 through FIG. 21, as the number of imagecandidates displayed in one operation guide 500 decreases, the imagecandidates can be displayed on a larger scale, thereby enabling the userto look into the image candidates in more detail before making theselection. For this reason, the image data of the image transformationdictionary part 43 can include a number of pixels capable of providingan excellent image quality in a state where the image data is displayedat a maximum size, and the image data can be displayed on a reducedscale, as appropriate, so that the image quality will not deterioratewhen displaying the image data on an enlarged scale.

FIG. 22 through FIG. 24 illustrate display examples of the operationguide 500 displaying the image candidate 531 and an image candidate name532. FIG. 22 corresponds to FIG. 19, FIG. 23 corresponds to FIG. 20, andFIG. 24 corresponds to FIG. 21. The display method illustrated in FIG.22 through FIG. 24 may be the same as that illustrated in FIG. 19through FIG. 21. In the examples illustrated in FIG. 22 through FIG. 24,the image transformation control part 42 also receives the name of theindex column together with the image, from the image transformationdictionary part 43, and transmits the same to the handwritten inputdisplay control part 23. Hence, the operation guide 500 can display thename of the image candidate.

Whether to display only the image candidates 531 as in FIG. 19 throughFIG. 21, or to display the image candidates 531 and the image candidatenames 532 as in FIG. 22 through FIG. 24, can be set by the user from anoperation header 520, or from a menu that is displayed by a long pressof the operation guide 500 or the like.

<Example of Specifying Selected Object>

The display device 2 according to this embodiment can specify theselected object when the user selects a decided object by handwriting.The selected object (or decided object) may be subject to editing ordecorating.

FIG. 25A through FIG. 25D are diagrams illustrating an example ofspecifying the selected object. In FIG. 25A through FIG. 25D, ahandwritten object 11 is displayed by a black solid line, a handwrittenobject rectangular region 12 is displayed by a gray halftone dotpattern, a decided object 13 is displayed by a black line, and aselected object rectangular region 14 is displayed by a dotted line.These objects and regions can be distinguished from one another by alowercase letter appended to the reference numeral designated thereto.Further, the striding line determination condition 406 or the enclosureline determination condition 407 of the defined control data illustratedin FIG. 7 can be used as a determination condition (whether or not apredetermined relationship is satisfied) for determining a decidedobject as the selected object.

FIG. 25A illustrates an example in which two decided objects 13 a and 13b written horizontally are specified by the user using the striding line(handwritten object 11 a). In this example, a length H1 of the shorterside and a length W1 of the longer side of a handwritten objectrectangular region 12 a satisfy the conditions of the striding linedetermination condition 406, and the overlap ratio of the handwrittenobject rectangular region 12 a with respect to the decided objects 13 aand 13 b, respectively, satisfies the conditions of the striding linedetermination condition 406. For this reason, both the decided objects13 a and 13 b that are the character string made up of three Kanjicharacters “

” pronounced “gijiroku” and the character string made up of two Hiraganacharacters “

” pronounced “giji”, respectively, are specified as the selectedobjects.

FIG. 25B illustrates an example in which a decided object 13 c writtenhorizontally is specified by the user using the enclosure line(handwritten object 11 b). In this example, only the decided object 13 cthat is the character string made up of three Kanji characters “

” pronounced “gijiroku”, is specified as the selected object, becausethe overlap ratio of the handwritten object rectangular region 12 c withrespect to the decided object 13 c satisfies the conditions of theenclosure line determination condition 407.

FIG. 25C illustrates an example in which a plurality of decided objects13 d and 13 e written vertically are specified by the user using thestriding line (handwritten object 11 c). In this example, similar toFIG. 25A, the length H1 of the shorter side and the length W1 of thelonger side of a handwritten object rectangular region 12 d satisfy theconditions of the striding line determination condition 406, and theoverlap ratio of the handwritten object rectangular region 12 c withrespect to the decided object 13 d that is the character string made upof three Kanji characters “

” pronounced “gijiroku”, and the decided object 13 e that is thecharacter string made up of two Hiragana characters “

” pronounced “giji”, respectively, satisfies the conditions of thestriding line determination condition 406. For this reason, the decidedobjects 13 d and 13 e of both the character string made up of threeKanji characters “

” pronounced “gijiroku” and the character string made up of two Hiraganacharacters “

” pronounced “giji”, are specified as the selected objects.

FIG. 25D illustrates an example in which a decided object 13 f isspecified by the user using the enclosure line (handwritten object lid).In this example, similar to FIG. 25B, only the decided object 13 f thatis the character string made up of three Kanji characters “

” pronounced “gijiroku” is specified as the selected object.

<Example of Displaying Operation Command Candidate>

FIG. 26A and FIG. 26B illustrate a display example of the operationcommand candidate based on the operation command definition data in acase where the handwritten object illustrated in FIG. 13 is present.FIG. 26A illustrates the operation command candidate for the editingsystem, and FIG. 26B illustrates the operation command candidate for thedecorating system. Further, FIG. 26A illustrates the example in whichthe selected object is specified by the handwritten object 11 aillustrated in FIG. 25A.

As illustrated in FIG. 26A and FIG. 26B, a main menu 550 displays theoperation command candidates after the bullet character “>>” 511. Themain menu 550 displays the last executed operation command name, or thefirst operation command name in the operation command definition data. Abullet character “>>” 511 a of the first line displays the operationcommand candidate for the editing system, and a bullet character “>>”511 b of the second line displays the operation command candidate forthe decorating system.

An end-of-line character “>” (an example of a sub menu button) in theoperation command 512 indicates that there is a sub menu. In the firstline, an end-of-line character “>” 512 a causes the (last selected) submenu to be displayed with respect to the operation command candidatesfor the editing system. In the second line, an end-of-line character “>”512 b causes remaining sub menus to be displayed with respect to theoperation command candidates for the decorating system. When the userpresses the end-of-line character “>” in the operation command 512, asub menu 560 is displayed on the right side thereof. The sub menu 560displays all operation commands defined in the operation commanddefinition data. In the display example illustrated in FIG. 26A, the submenu 560 corresponding to the end-of-line character “>” 512 a of thefirst line is also displayed from the time when the main menu 550 isdisplayed. The sub menu 560 may be displayed when the user presses theend-of-line character “>” 512 a of the first line.

When the user presses one of the operation command names by using thepen, the handwritten input display control part 23 executes the“Command” of the operation command definition data corresponding to theoperation command name, with respect to the selected object. In otherwords, “Delete” is executed when a “Delete” button 521 is selected,“Move” is executed when a “Move” button 522 is selected, “Rotate” isexecuted when a “Rotate” button 523 is selected, and “Select” isexecuted when a “Select” button 524 is selected.

For example, if the user presses the “Delete” button 521 with the pen,the character string made up of three Kanji characters “

” pronounced “gijiroku” and the character string made up of the twoHiragana characters “

” pronounced “giji” can be deleted. Pressing the “Move” button 522, the“Rotate” button 523, and the “Select” button 524 causes a bounding box(circumscribed rectangle of the selected object). The “Move” button 522and the “Rotate” button 523 allows the user to move or rotate thecharacters by a drag operation of the pen. Pressing the “Select” button524 allows the user to perform other bounding box operations.

Character string candidates other than the operation command candidates,such as “-” 541, “-” 542, “˜” 543, “→” 544, and “⇒” 545, are therecognition results of the striding line (handwritten object 11 a).Hence, if the user intends to input the character string and not theoperation command, the character string candidate can be selected.

In FIG. 26B, when the user presses the end-of-line character “>” 512 bof the second line, the sub menu 560 is displayed on the right sidethereof. Similar to FIG. 26A, FIG. 26B illustrates the example in whichboth the main menu 550 and the sub menu 560 are displayed. When “Thick”531 a is selected based on the operation command definition dataillustrated in FIG. 13, the handwritten input display control part 23executes “Thick” on the selected object to make the selected objectthick. When “Thin” 532 a is selected, the handwritten input displaycontrol part 23 executes “Thin” with respect to the selected object tomake the selected object thin. When “Large” 533 a is selected, thehandwritten input display control part 23 executes “Large” with respectto the selected object to make the selected object large. When “Small”534 a is selected, the handwritten input display control part 23executes “Small” with respect to the selected object to make theselected object small. When “Underline” 535 a is selected, thehandwritten input display control part 23 executes “Underline” withrespect to the selected object to underline the selected object.

Fixed or default values may be defined separately with respect to theextent to which the selected object is to be thickened when “Thick” 531a is selected, the extent to which the selected object is to be thinnedwhen “Thin” 532 a is selected, the extent to which the selected objectis to be enlarged when “Large” 533 a is selected, the extent to whichthe selected object is to be reduced when “Small” 534 a is selected, andthe line type to be used when “Underline” 535 a is selected, or thelike. Alternatively, when the sub menu illustrated in FIG. 26B isselected, a separate selection menu can be opened to allow the user tomake adjustments to the selected object.

When the user presses “Thick” 531 a with the pen, the handwritten inputdisplay control part 23 thickens the lines forming the decided objects13 a and 13 b that are the character string made up of three Kanjicharacters “

” pronounced “gijiroku” and the character string made up of two Hiraganacharacters “

” pronounced “giji”, respectively. When the user presses “Thin” 532 awith the pen, the handwritten input display control part 23 thins thelines forming the decided objects 13 a and 13 b that are the characterstring made up of three Kanji characters “

” pronounced “gijiroku” and the character string made up of two Hiraganacharacters “

” pronounced “giji”, respectively. When the user presses large 533 awith the pen, the handwritten input display control part 23 can enlargethe decided objects 13 a and 13 b, respectively. When the user presses“Small” 534 a with the pen, the handwritten input display control part23 can reduce the decided objects 13 a and 13 b, respectively. When theuser presses “Underline” 535 a with the pen, the handwritten inputdisplay control part 23 can add the underline to the decided objects 13a and 13 b, respectively.

FIG. 27A and FIG. 27B illustrate a display example of the operationcommand candidate based on the operation command definition data whenthe handwritten object illustrated in FIG. 13 is present. The differencefrom FIG. 26A and FIG. 26B is that FIG. 27A and FIG. 27B illustrate theexample in which the selected object is specified by the handwrittenobject 11 b (enclosure line) illustrated in FIG. 25B. As may be seen bycomparing FIG. 26A and FIG. 26B with FIG. 27A and FIG. 27B, there is nodifference in the operation command candidates that are displayed,regardless of whether the handwritten object is the line or theenclosure line, so to enable the handwritten input display control part23 to display the operation command candidate on the display part 22when the selected object is specified. However, the handwritten inputdisplay control part 23 may recognize the handwritten object and changethe operation command candidates according to the handwritten object. Inthis case, a developer or the like associates the operation commanddefinition data such as that illustrated in FIG. 13 with the recognizedhandwritten objects (“-”, “o”, or the like), so as to providecorrespondence between the recognized handwritten objects and theoperation command definition data.

In FIG. 27A and FIG. 27B, character string candidates other than theoperation command candidates, namely, “∘” 551, “∞” 552, “0” 553, “00”554, and “

” 555, are the recognition results of the enclosure line (handwrittenobject 11 b), and the character string candidate can be selected if theuser intends to input the character string and not the operationcommand. “

” 555 is a Katakana character pronounced “ro”.

<Input Example of Angular Information>

Next, a method of inputting angular information will be described, withreference to FIG. 28A through FIG. 28C. FIG. 28A through FIG. 28Cillustrate an example of the input method of angular information. FIG.28A through FIG. 28C illustrate a case where the user, positioned in the3 o'clock direction from the display device 2, inputs the angularinformation. Because handwritten characters or the like written from the3 o'clock direction can be recognized correctly when rotated 90 degreesclockwise, the angular information indicating 90 degrees may be input.

FIG. 28A illustrates a state in which the angular information of the penID control data is 0 degrees (initial value), and the operation guide500 is displayed because the user, positioned in the 3 o'clock directionfrom the display device 2, handwrites the Hiragana character “

” pronounced “gi”. Because the display device 2 recognizes the Hiraganacharacter “

” pronounced “gi”, handwritten from the 3 o'clock direction, with theangular information maintained at 0 degrees, the selectable candidate530 that is different from the anticipated selectable candidate isdisplayed.

When inputting the angular information, the user handwrites a straightline inside the operation guide 500, from the top to bottom when viewedfrom the user. FIG. 28B illustrates an example of a straight line 5211.The angular information is an angle α formed in a counterclockwisedirection between the straight line 5211 and a 6 o'clock direction(angular information of 0 degrees). In other words, the angularinformation is the angle α formed in the counterclockwise directionbetween the straight line 5211 input by the user and a straight line5221 (indicated by a dotted line) that is lowered in the 6 o'clockdirection from a starting point S of the straight line 5211. Simply put,the angular information is an end point direction of the straight line5211. Accordingly, the angular information input by the user in FIG. 28Bis 90 degrees.

A method of detecting the straight line may convert coordinates from thestarting point S to an end point E into a straight line according to themethod of least squares, and comparing a correlation coefficient with athreshold value to determine whether or not the straight line isobtained, for example.

Immediately after the user starts writing the straight line 5211(immediately after the pen 2500 touches the starting point S of thestraight line 5211), the display device 2 deletes the operation guide500. In addition, immediately after the writing of the straight line5211 is completed (immediately after the pen 2500 is separated from theend point E of the straight line 5211), the display device 2 searchesfor the value closest to the above described angle α from 45 degrees, 90degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315degrees, and 360 degrees, to determines the closest value as the angularinformation. The angle α itself may be the angular information. Thedetermined angular information is set to “Angle” of the pen ID controldata. The pen event transmitting part 41 of the pen 2500 transmits thepen ID to the display device 2 when the pen tip of the pen 2500 ispressed for the handwriting or the like, and thus, the display device 2can make the angular information correspond to the pen ID control data.

The user can handwrite the straight line to input the angularinformation, only from the operation guide 500. Accordingly, when theuser handwrites the straight line in regions other than the operationguide 500, the straight line is recognized as a numeric character “1” ora Kanji character “

” pronounced “ichi” (“one” in English) or the like, and the angularinformation can be input when the straight line is handwritten in theoperation guide 500. In other words, the handwriting recognition controlpart 26 detects the straight line from a predetermined range, andconverts the handwritten stroke data outside the predetermined rangeinto the text data.

FIG. 28C illustrates the operation guide 500 immediately after theoperation illustrated in FIG. 28B is performed. Because 90 degrees isset as the angular information (“Angle”) in the pen ID control data, thehandwritten object (stroke data) is internally rotated clockwise by 90degrees to be subject to the handwriting recognition, and the operationguide 500 is rotated counterclockwise by 90 degrees and displayed.

FIG. 29 is a diagram for explaining another method of inputting theangular information. In FIG. 29, it is assumed for the sake ofconvenience that the user is positioned in the 3 o'clock direction fromthe display device 2. In FIG. 29, because the user positioned in thedirection of 3 o'clock from the display device 2 handwrites the Hiraganacharacter “

” pronounced “gi” in the state where the angular information is 0degrees (initial value), the operation guide 500 and the selectablecandidate 530 are displayed. The operation guide 500 illustrated in FIG.29 includes a rotating operation button 5111 in the operation header520.

The angular information of the pen ID control data is incremented by 90degrees every time the user presses the rotating operation button 5111using the pen 2500, to change the angular information to a remainderthat is obtained after dividing the incremented angular information by360 degrees. The angular information may be incremented by 45 degreesevery time the rotating operation button 5111 is pressed.

<Registration Example of Handwritten Signature Data>

Next, a registration example of the handwritten signature data will bedescribed, with reference to FIG. 30A through FIG. 30C. FIG. 30A throughFIG. 30C are diagrams for explaining a method of registering thehandwritten signature data. First, FIG. 30A illustrates an example ofthe selectable candidate 530 that is displayed when the user handwritesa character string made up of three Katakana characters “

” pronounced “sain” (“signature” in English). In this example, twooperation commands 513 and 514 for a character string “

” pronounced “tegaki sain touroku suru” (“register handwrittensignature” in English) and a character string “

” pronounced “tegaki sain auto suru” (“handwritten sign out” inEnglish), respectively, based on the operation command definition data713 and 715, and the character candidates made up of the characterstring “

” pronounced “sain” (“signature” in English), the character string “

” pronounced “sain kai” (“autograph session” in English), and thecharacter string “

” pronounced “sain iri” (“autographed” in English), partially matchingthe character string made up of the three Katakana characters “

” pronounced “sain” (“signature” in English), are displayed. The twooperation commands 513 and 514 are displayed because “String” of theoperation command definition data 713 and 715 illustrated in FIG. 12Aincludes the character string made up of the three Katakana characters “

” pronounced “sain” (“signature” in English).

When the user presses the operation command 513 for the character string“

” pronounced “tegaki sain touroku suru” (“register handwrittensignature” in English) by the pen 2500, a handwritten signatureregistration form 561 illustrated in FIG. 30B is added to thehandwritten input storage part 25 and is displayed on the operationscreen 101. For example, the operation guide 500 illustrated in FIG. 30Ais deleted, and the handwritten signature registration form 561 isdisplayed in the same position as the operation guide 500. Thehandwritten signature registration form 561 includes, from the top, aname input field 561 a, signature input fields 561 b through 561 d, anda registration confirmation field 561 e. The user inputs the text of thename in the name input field 561 a, a first-time handwritten signature,a second-time handwritten signature, and a third-time handwrittensignature in the signature input fields 561 b, 561 c, and 561 d,respectively, and a check mark or a cancellation mark in theregistration confirmation field 561 e. The text of the name is thedisplay name of this user, and is converted into text data. Thehandwritten signatures are input three times because feature quantitiesof the handwritten signatures are registered on the precondition that,the handwritten signature is different each time the user handwrites thesignature and a perfect match of the handwritten signatures of the sameuser rarely occurs.

Generally, the handwritten signature is the user's name or the like, orcharacters or the like related to the user. Besides the user's name, thehandwritten signature may be a number such as the employee number, anickname, a portrait, or the like. In addition, the handwrittensignature is not limited to the characters or the like related to theuser, and may be some kind of handwritten object. The handwrittensignature may be a circle, a triangle, a square, a symbol, or anarbitrary combination thereof. Because feature data of the handwrittensignature are not limited to the coordinates thereof, even if two usershaving the same surname (for example, “Suzuki”) register the handwrittensignatures for “Suzuki” using the same three Hiragana characters “

” it is possible to correctly authenticate the two users.

When the user handwrites on the handwritten signature registration form561 according to instructions, the handwritten signature registrationform 561 as illustrated in FIG. 30C is obtained. When the userhandwrites a character string made up of seven Katakana characters “

” pronounced “chekku maaku” (“check mark” in English) in theregistration confirmation field 561 e, the handwritten signature data isregistered in the handwritten signature data storage part 39, and thehandwritten signature registration form 561 is deleted. By thisregistration, “SignatureId” is numbered, and the similarly numbered“AccountId”, and the text indicating the name of the name input field561 a are registered in the user definition data in correspondence with“SignatureId”. When the user handwrites the user's name or the like andsigns in, “SignatureId” corresponding to “AccountId” in the userdefinition data is acquired, and registered in the pen ID control datain correspondence with the pen ID of the pen 2500 that is used for thehandwritten sign in. Thereafter, if the user uses the pen 2500, the penID is transmitted to the display device 2. The display device 2 canidentify “AccountId” corresponding to the pen ID in the pen ID controldata, and can execute the operation command using the user definitiondata without requiring user intervention.

When the user handwrites “x” in the registration confirmation field 561e, the handwritten signature registration is cancelled, and thehandwritten signature registration form 561 is deleted. If an erroroccurs during the registration, the error is displayed in a systemreservation area or the like of the operation screen 101.

As described above, the handwritten input display control part 23 canaccept the handwritten input without distinguishing between thehandwritten input to the form and the handwritten input to other thanthe form.

<Example of Handwritten Sign In>

Next, a method employed by the user to sign in after registering thehandwritten signature data will be described, with reference to FIG. 31.FIG. 31 illustrates an example of the operation guide 500 that isdisplayed when the user handwrites a character string “

” (“Suzuki” in English) which is the handwritten signature dataregistered by the user. Because the character string “

” is registered in the operation command definition part 33 as thehandwritten signature data, the character string “

” handwritten by the user meets the criteria for the match with theregistered handwritten signature data. Accordingly, the operationcommand 512, which is a character string of eight Kanji, Hiragana, andKatakana characters “

” pronounced “tegaki sain suru” (“Affix handwritten signature” inEnglish) is displayed.

In addition, because handwritten signature data meets the criteria forthe match with the registered handwritten signature data, “SignatureId”representing the character string “

” (“Suzuki” in English) is identified, and the user definition datahaving “AccountId” corresponding to “SignatureId” is identified.

If the user selects the operation command 512, which is a characterstring of eight Kanji, Hiragana, and Katakana characters “

” pronounced “tegaki sain suru” (“Affix handwritten signature” inEnglish), the user definition data of the character string “

” (“Suzuki” in English) is added to the pen ID control data incorrespondence with the pen ID of the pen 2500 that is being used by theuser, so that the user definition data of the character string “

” (“Suzuki” in English) can be used when using the operation command.

Because the registration of the handwritten signature data using thehandwritten signature registration form 561 illustrated in FIG. 31 iscontrolled as part of the handwritten input represented by thecharacters or the like, the handwritten signature registration form 561is displayed on the same operation screen as the operation screen onwhich the characters or the like are handwritten. There is no differencein the handwriting operations inside and outside the handwrittensignature registration form 561, and the user can complete the input tothe handwritten signature registration form 561 by simply handwriting toregions of the handwritten signature registration form 561 partitionedby ruled lines.

<Example of User Definition Data Changing Operation>

Next, a method of changing the user definition data will be described,with reference to FIG. 32A and FIG. 32B. FIG. 32A and FIG. 32B arediagrams for explaining the method of changing the user definition data.FIG. 32A illustrates an example of the operation guide 500 that isdisplayed when the user handwrites a Hiragana character “

” pronounced “se”. In the operation command definition data 716illustrated in FIG. 12A, a character string made up of two Kanjicharacters “

” pronounced “settei” (“set” or “setting” in English) is defined in“String”, and because the character string “

” is included in the predicted character strings for the Hiraganacharacter “

”, the operation command 512, which is a character string made up of sixKanji and Hiragana characters “

” pronounced “settei henkou suru” (“change setting” in English), isdisplayed.

If the user selects the operation command 512, which is the characterstring “

” pronounced “settei henkou suru” (“change setting” in English), withthe pen 2500 that is used to handwrite the signature, the “AccountId” ofthe user is identified from the pen ID control data corresponding to thepen ID of the pen 2500. Hence, the user definition data of the user whosigned in is identified. A user definition data change form 562illustrated in FIG. 32B is added to the handwritten input storage part25, and is displayed on the operation screen 101. In the exampleillustrated in FIG. 32B, the user definition data change form 562 iscreated in according to the user definition data 718 illustrated in FIG.14. The user definition data change form 562 includes a name field 562a, a password field 562 b, a folder user name field 562 c, a folderpassword field 562 d, a folder file name field 562 e, and a registrationor cancellation field 562 f.

If the user has not signed in beforehand, an error occurs because thedisplay device 2 cannot identify the “AccountId” of the user, and anerror message is displayed in the system reservation area or the like ofthe operation screen 101.

The user handwrites the password in the password field 562 b of the userdefinition data change form 562 illustrated in FIG. 32B. The userhandwrites the folder user name in the folder user name field 562 c. Theuser handwrites the folder password in the folder password field 562 d,and handwrites the folder file name in the folder file name field 562 e.The user handwrites a check mark “√” or “x” in the registration orcancellation field 562 f. As a result of these inputs to the userdefinition data change form 562, changes are made to the user definitiondata, and the user definition data change form 562 is then deleted.

Accordingly, the user can cause the display of the user definition datachange form 562 by handwriting the stroke data that calls and makesaccess to the user definition data change form 562, and the user canarbitrarily change the user definition data by inputting the changes tothe user definition data change form 562. The handwritten input displaycontrol part 23 receives the handwritten input without distinguishingbetween the handwritten input to the form and the handwritten input toother than the form.

The “AccountUsername” of the user definition data is automaticallydisplayed in the name field 562 a. In addition, the user definition datachange form 562 can be used for not only for making changes, but alsofor making registrations.

Because the change of the user definition data using the user definitiondata change form 562 illustrated in FIG. 32B is controlled as part ofthe handwritten input typified by characters or the like, the userdefinition data change form 562 is displayed on the same operationscreen as the operation screen on which the characters or the like arehandwritten. There is no difference in the handwriting operations insideand outside the user definition data change form 562, and the user cancomplete the input to the user definition data change form 562 by simplyhandwriting to regions of the user definition data change form 562partitioned by ruled lines.

<Operation Procedure>

The operation of the display device will be described using the abovedescribed configurations, with reference to FIG. 33 through FIG. 39.FIG. 33 through FIG. 39 are sequence diagrams illustrating an example ofa process in which the display device 2 displays the character stringcandidate and the operation command candidate. The process illustratedin FIG. 33 starts when the display device 2 is started (when theapplication program is started). In FIG. 33 through FIG. 39, thefunctions illustrated in FIG. 6A are indicated by the reference numeralsfor the sake of convenience, due to space limitations.

First, in step S1 illustrated in FIG. 33, the handwritten input displaycontrol part 23 transmits the start of the handwritten object to thehandwritten input storage part 25, and in response thereto, thehandwritten input storage part 25 secures a handwritten object region (amemory region for storing the handwritten object). The handwrittenobject region may be secured after the user causes the pen 2500 to makecontact with the handwritten input part 21.

Next, in step S2, the user causes the pen 2500 to make contact with thehandwritten input part 21, and the handwritten input part 21 detects andtransmits the pen down to the handwritten input display control part 23.

In step S3, the handwritten input display control part 23 transmits astart of the stroke to the handwritten input storage part 25, and thehandwritten input storage part 25 secures a stroke region.

In step S4, when the user moves the pen 2500 while the pen 2500maintains contact with the handwritten input part 21, the handwritteninput part 21 transmits the pen coordinates to the handwritten inputdisplay control part 23.

In step S5, the handwritten input display control part 23 specifies thepen ID received from the pen 2500 at the same time as the input of thepen coordinates, and acquires the current pen ID control data stored inthe pen ID control data storage part 36. Because the pen ID istransmitted at the time of inputting the pen coordinates, the stroke andthe pen ID are made to correspond to each other. The pen ID control datastorage part 36 transmits the pen ID control data (color, thickness,pattern, and angular information) to the handwritten input displaycontrol part 23. In this state, the angular information still has theinitial value which is zero.

In step S6, the handwritten input display control part 23 transmits pencoordinate complement display data (data interpolating discrete pencoordinates) to the display part 22. The display part 22 displays a lineby interpolating the pen coordinates using the pen coordinate complementdisplay data.

In step S7, the handwritten input display control part 23 transmits thepen coordinates and a reception time thereof to the handwritten inputstorage part 25, and the handwritten input storage part 25 adds the pencoordinates to the stroke. While the user is moving the pen 2500, thehandwritten input part 21 periodically repeats transmitting the pencoordinates to the handwritten input display control part 23, and thus,the processes of steps S4 through S7 are repeated until the pen up.

In step S8 illustrated in FIG. 34, when the user releases the pen 2500from the handwritten input part 21, the handwritten input part 21transmits the pen up to the handwritten input display control part 23.

In step S9, the handwritten input display control part 23 transmits anend of the stroke to the handwritten input storage part 25, and thehandwritten input storage part 25 determines the pen coordinates of thestroke. The pen coordinates cannot be added to the stroke after the pencoordinates of the stroke are determined.

Next, in step S10, the handwritten input display control part 23transmits an overlapping state acquisition of the handwritten objectrectangular region and the stroke rectangular region to the handwritteninput storage part 25, based on the handwritten object rectangularregion 403. The handwritten input storage part 25 calculates theoverlapping state, and transmits the calculated overlapping state to thehandwritten input display control part 23.

Subsequent steps S11 through S17 are performed when the handwrittenobject rectangular region and the stroke rectangular region do notoverlap each other.

In step S11, if the handwritten object rectangular region and the strokerectangular region do not overlap each other, one handwritten object isdetermined, and thus, the handwritten input display control part 23transmits a stored data clear to the handwriting recognition controlpart 26.

In steps S12 through S14, the handwriting recognition control part 26transmits the stored data clear to each of the character stringconversion control part 28, the predictive conversion control part 30,and the operation command recognition control part 32. In response tothe stored data clear, the handwriting recognition control part 26, thecharacter string conversion control part 28, the predictive conversioncontrol part 30, and the operation command recognition control part 32clear the data related to the character string candidates and theoperation command candidates stored up to a point in time immediatelybefore receiving the stored data clear. At the time of clearing thedata, the last handwritten stroke is not added to the handwrittenobject.

In step S15, the handwritten input display control part 23 transmits theend of the handwritten object to the handwritten input storage part 25,and the handwritten input storage part 25 determines the handwrittenobject. The handwritten object is determined when one handwritten objectis completed (no more strokes are added).

In step S16, the handwritten input display control part 23 transmits thestart of the handwritten object to the handwritten input storage part25. In order to prepare for the start of handwriting (pen down) of thenext handwritten object, the handwritten input storage part 25 secures anew handwritten object region.

Next, in step S17, the handwritten input display control part 23transmits a stroke addition with respect to the stroke ended in step S9to the handwritten input storage part 25. When steps S11 through S17 areperformed, the added stroke is the first stroke of the handwrittenobject, and the handwritten input storage part 25 adds the stroke datato the handwritten object that is being started to be handwritten. Ifsteps S11 through S17 are not performed, the added stroke is alreadyadded to the handwritten objects that is being handwritten.

Subsequently, in step S18 illustrated in FIG. 35, the handwritten inputdisplay control part 23 transmits the stroke addition to the handwritingrecognition control part 26, and the handwriting recognition controlpart 26 adds stroke data to a stroke data storage region (a region wherethe stroke data is temporarily stored) where the character stringcandidates are stored.

In step S19, the handwriting recognition control part 26 performs agesture handwriting recognition with respect to the stroke data storageregion. The gesture handwriting recognition refers to the recognition ofthe angular information from the straight line. Because the gesturehandwriting recognition is performed inside the operation guide 500, thehandwriting recognition control part 26 detects the straight line insidethe operation guide 500. The position information of the operation guide500 is transmitted to the handwriting recognition control part 26 instep S67, which will be described later.

In step S20, when the straight line inside the operation guide 500 isdetected, the angle α formed in the counterclockwise direction betweenthe straight line 5211 input by the user and the straight line 5221 thatis lowered in the 6 o'clock direction from the starting point S of thestraight line 5211, is determined in units of 45 degrees. In addition,the handwriting recognition control part 26 stores the determinedangular information in the pen ID control data storage part 36 incorrespondence with the pen ID of the stroke data of the straight line5211. Step S20 is performed when the straight line is detected in theoperation guide 500.

Next, in step S21, the handwriting recognition control part 26 specifiesthe pen ID received from the handwritten input part 21, and acquires theangular information of the current pen ID control data from the pen IDcontrol data storage part 36.

In step S22, the handwriting recognition control part 26 rotates thestroke data of the stroke data storage region clockwise according to theacquired angular information.

In step S23, the handwriting recognition control part 26 transmits thestroke data after rotation, namely, the rotated stroke data, to thehandwritten signature authentication control part 38. Hence, the strokedata is always transmitted to the handwritten signature authenticationcontrol part 38 in a state where it is unclear whether or not the strokedata is the handwritten signature.

In step S24, the handwritten signature authentication control part 38receives the stroke data, and receives the registered handwrittensignature data from the handwritten signature data storage part 39.Further, the handwritten signature authentication control part 38compares the stroke data with the handwritten signature data todetermine a match, and stores the authentication result of thehandwritten signature so that the authentication result of thehandwritten signature can be acquired in step S61 at a subsequent stagewhich will be described later.

Next, in step S25, the handwriting recognition control part 26 performsa handwriting recognition on the stroke data, and processes the formwhen the checkmark “q” or “x” is recognized in the registration orcancellation field of the form, and otherwise performs the conventionalhandwriting recognition.

In step S26, when the registration or cancellation field of thehandwritten signature registration form 561 includes the check mark “q”,the handwriting recognition control part 26 transmits the handwrittensignature data (stroke data) input by the user with respect to thehandwritten signature registration form 561 to the handwritten signatureauthentication control part 38. The handwritten signature registrationform 561 is generated in the handwritten input storage part 25 by thehandwritten input display control part 23 in step S86, which will bedescribed later.

In step S27, the handwritten signature authentication control part 38registers the received handwritten signature data (stroke data) in thehandwritten signature data storage part 39, and “SignatureId” isnumbered and returned to the handwriting recognition control part 26. If“SignatureId” and the name input in the name input field 561 a of thehandwritten signature registration form 561 are not stored in the userdefinition data, the handwriting recognition control part 26 newly addsthe user definition data. In addition, the handwriting recognitioncontrol part 26 assigns numbering to “AccountId”, and stores“SignatureId” in the added user definition data. If the name input inthe name input field 561 a is stored in the user definition data,“SignatureId” is stored in the user definition data. This process links“AccountId” and “SignatureId”. When the user definition data is newlyadded, other values are not set, but the user can register and changeother values from the user definition data change form 562.

In step S28, upon registration of the handwritten signature data, thehandwriting recognition control part 26 deletes the handwrittensignature registration form 561 from the handwritten input storage part25.

In step S29, when the registration or cancellation field of the userdefinition data change form 562 includes the check mark “q”, thehandwriting recognition control part 26 transmits the changed valueinput to the user definition data change form 562 in step S86, whichwill be described later, to the operation command definition part 33.The handwritten input display control part 23 generates the userdefinition data change form 562 in the handwritten input storage part25.

In step S30, upon changing the user definition data, the handwritingrecognition control part 26 deletes the user definition data change form562 from the handwritten input storage part 25.

In step S31, when the registration or cancellation field of the formadded in step S86, which will be described later, includes a mark “x”,the handwriting recognition control part 26 deletes the form added instep S86 from the handwritten input storage part 25.

In step S33 illustrated in FIG. 36, if the form is not processed, thehandwriting recognition control part 26 transmits, as the processedresult, the recognized handwritten character string candidates to thehandwriting recognition dictionary part 27. The handwriting recognitiondictionary part 27 transmits the language character string candidatesthat are linguistically probable to the handwriting recognition controlpart 26.

In step S34, the handwriting recognition control part 26 transmits therecognized handwritten character string candidates and the receivedlanguage character string candidates to the character string conversioncontrol part 28.

In step S35, the character string conversion control part 28 transmitsthe recognized handwritten character string candidates and the languagecharacter string candidates to the character string conversiondictionary part 29. The character string conversion dictionary part 29transmits the converted character string candidates to the characterstring conversion control part 28.

In step S36, the character string conversion control part 28 transmitsthe received converted character string candidates to the predictiveconversion control part 30.

In step S37, the predictive conversion control part 30 transmits thereceived converted character string candidates to the predictiveconversion dictionary part 31. The predictive conversion dictionary part31 transmits the predicted character string candidates to the predictiveconversion control part 30.

In step S38, the predictive conversion control part 30 transmits thereceived predicted character string candidates to the operation commandrecognition control part 32.

In step S39, the operation command recognition control part 32 transmitsthe received predicted character string candidates to the operationcommand definition part 33. The operation command definition part 33transmits the operation command candidates to the operation commandrecognition control part 32. Accordingly, the operation commandrecognition control part 32 can acquire the operation command candidatecorresponding to the operation command definition data including thecharacter string (String) matching the predicted character stringcandidate.

Thereafter, the processes up to the transmission of the operationcommand candidates described in conjunction with steps S40 through S47are performed similarly.

In step S40, the character string conversion control part 28 transmitsthe received converted character string candidates to the operationcommand recognition control part 32.

In step S41, the operation command recognition control part 32 transmitsthe received converted character string candidates to the operationcommand definition part 33. The operation command definition part 33transmits the operation command candidates to the operation commandrecognition control part 32. Accordingly, the operation commandrecognition control part 32 can acquire the operation command candidatecorresponding to the operation command definition data including thecharacter string (String) matching the converted character stringcandidate.

In step S42, the handwriting recognition control part 26 transmits therecognized handwritten character string candidates and the languagecharacter string candidates to the predictive conversion control part30.

In step S43, the predictive conversion control part 30 transmits therecognized handwritten character string candidates and the receivedlanguage character string candidates to the predictive conversiondictionary part 31. The predictive conversion dictionary part 31transmits the predicted character string candidates to the predictiveconversion control part 30.

In step S44, the predictive conversion control part 30 transmits thereceived predicted character string candidates to the operation commandrecognition control part 32.

In step S45, the operation command recognition control part 32 transmitsthe received predicted character string candidates to the operationcommand definition part 33. The operation command definition part 33transmits the operation command candidates to the operation commandrecognition control part 32. Accordingly, the operation commandrecognition control part 32 can acquire the operation command candidatecorresponding to the operation command definition data including thecharacter string (String) matching the predicted character stringcandidate.

In step S46, the handwriting recognition control part 26 transmits thehandwritten identification character string candidates and the receivedlanguage character string candidates to the operation commandrecognition control part 32.

In step S47, the operation command recognition control part 32 transmitsthe recognized handwritten character string candidates and the receivedlanguage character string candidates to the operation command definitionpart 33. The operation command definition part 33 transmits theoperation command candidates to the operation command recognitioncontrol part 32. Accordingly, the operation command recognition controlpart 32 can acquire the operation command candidate corresponding to theoperation command definition data including the character string(String) matching the language character string candidate.

Next, in step S48, the handwriting recognition control part 26 transmitsthe stroke addition to the operation command recognition control part32.

In step S49, the operation command recognition control part 32 transmitsthe position information acquisition of the decided object to thehandwritten input storage part 25. The handwritten input storage part 25transmits the position information of the decided object to theoperation command recognition control part 32.

In step S50, the operation command recognition control part 32determines whether or not the position information of the strokereceived from the handwriting recognition control part 26 by the strokeaddition in step S48 is in a predetermined relationship with theposition information of the decided object received from the handwritteninput storage part 25, based on the striding line determinationcondition 406 and the enclosure line determination condition 407, inorder to determine the selected object. The operation commandrecognition control part 32 stores the decided object that can bedetermined to be selected, as the selected object. In this case, becausethe selected object is identified, the operation command candidates ofthe input and output system are acquired from the operation commanddefinition part 33.

Further, the handwriting recognition control part 26, the characterstring conversion control part 28, the predictive conversion controlpart 30, and the operation command recognition control part 32 store thedata related to the recognized handwritten character string candidates,the language character string candidates, the converted character stringcandidates, the predicted character string candidates, the operationcommand candidates, and the selected object, so that the data can beacquired in steps S55 through S58 at subsequent stages which will bedescribed later, respectively.

In step S18-2, the handwritten input display control part 23 transmitsthe start of the selectable candidate display timer to the candidatedisplay timer control part 24, immediately after transmitting the strokeaddition to the handwriting recognition control part 26 in step S18. Thecandidate display timer control part 24 starts the selectable candidatedisplay timer in response to receiving the start of the selectablecandidate display timer.

Subsequent steps S51 through S53 are performed if the pen down occursbefore a predetermined time elapses (before the time out of the timeroccurs).

In step S51 illustrated in FIG. 37, if the user causes the pen 2500 tocontact the handwritten input part 21 before the time out of the timeroccurs, the handwritten input part 21 transmits the pen down (the sameevent as in step S2) to the handwritten input display control part 23.

In step S52, the handwritten input display control part 23 transmits thestart of the stroke (the same as in step S3) to the handwritten inputstorage part 25. The sequence after step S52 is the same as the sequenceafter step S3.

In step S53, the handwritten input display control part 23 transmits theselectable candidate display timer stop request to the candidate displaytimer control part 24. The candidate display timer control part 24 stopsthe selectable candidate display timer in response to the stop request,because the pen down is detected, thereby eliminating the need fortimer.

Steps S54 through S89 are performed if no pen down occurs before apredetermined time elapses (before the timer times out). Accordingly,the operation guide 500 illustrated in FIG. 18 is displayed.

In step S54, the candidate display timer control part 24 transmits thetime out to the handwritten input display control part 23 if the userdoes not cause the pen 2500 to contact the handwritten input part 21after the selectable candidate display timer is started.

In step S55, the handwritten input display control part 23 transmits theacquisition request of the handwriting recognition characterstring/language character string candidates to the handwritingrecognition control part 26. In response to this acquisition request,the handwriting recognition control part 26 transmits the handwritingrecognition character string/language character string candidatescurrently stored to the handwritten input display control part 23.

In step S56, the handwritten input display control part 23 transmits theacquisition request for the converted character string candidates to thecharacter string conversion control part 28. In response to thisacquisition request, the character string conversion control part 28transmits the currently stored converted character string candidates tothe handwritten input display control part 23.

In step S57, the handwritten input display control part 23 transmits theacquisition request for the predicted character candidates to thepredictive conversion control part 30. In response to this acquisitionrequest, the predictive conversion control part 30 transmits thepredicted character string candidates currently stored to thehandwritten input display control part 23.

In step S58, the handwritten input display control part 23 transmits theacquisition request for the operation command candidates to theoperation command recognition control part 32. In response to thisacquisition request, the operation command recognition control part 32transmits the currently stored operation command candidates and selectedobjects to the handwritten input display control part 23.

Next, in step S58-2, the handwritten input display control part 23specifies the recognized handwritten character string candidates and thelanguage character string candidates acquired from the handwritingrecognition control part 26, the converted character string candidatesacquired from the character string conversion control part 28, and thepredicted character string candidates acquired from the predictiveconversion control part 30, and transmits the image acquisition requestto the image transformation control part 42.

In step S58-3, the image transformation control part 42 performs thesearch in the image transformation dictionary part 43 using thehandwriting recognition character string candidates, the languagecharacter string candidates, the converted character string candidates,and the predicted character string candidates. Accordingly, the imagetransformation control part 42 receives the image candidates, andtransmits the received image candidates to the handwritten input displaycontrol part 23. The handwritten input display control part 23 mayreceive the names in addition to the image candidates.

Further, in step S59, the handwritten input display control part 23transmits the acquisition request for the estimated writing direction tothe handwritten input storage part 25. In response to this acquisitionrequest, the handwritten input storage part 25 determines the estimatedwriting direction from a stroke addition time, the horizontal distance,and the vertical distance of the handwritten object rectangular region,and transmits the estimated writing direction to the handwritten inputdisplay control part 23.

Next, in step S60, the handwritten input display control part 23specifies the pen ID received from the handwritten input part 21, andacquires the angular information of the current pen ID control data fromthe pen ID control data storage part 36.

In step S61, the handwritten input display control part 23 acquires theauthentication result of the handwritten signature from the handwrittensignature authentication control part 38, to enable “SignatureId” of theuser to be obtained. Hence, “AccountId” is registered in correspondencewith the pen ID when executing the operation command, as will bedescribed later.

In step S62, the handwritten input display control part 23 creates theselectable candidate display data, such as those illustrated in FIG. 18,from the recognized handwritten character string candidates (“

” in FIG. 18), the language character string candidates (not displayedin FIG. 18, but may be “

”, for example), the converted character string candidates (“

” and “

” in FIG. 18), the predicted character string candidates (“

” and “

” in FIG. 18), the operation command candidates (“

” and “

” in FIG. 18), each of the probabilities of selection, and the estimatedwriting direction.

Moreover, the handwritten input display control part 23 creates theselectable candidate display data including the image candidatesillustrated in FIG. 19 through FIG. 24. The image candidates and/or thecharacter string candidates of the selectable candidate display data tobe displayed depends on the user's settings, and for this reason, if theuser's settings selects the display of the image candidates, theselectable candidate display data including only the character stringcandidates do not need to be created.

Further, the handwritten input display control part 23 rotates theselectable candidate display data (the operation guide 500)counterclockwise based on the angular information acquired in step S60,and displays the rotated selectable candidate display data (theoperation guide 500) after rotation to the display part 22.

In step S63, the handwritten input display control part 23 rotates therectangular area display data (rectangular frame) of the handwrittenobject and the selected object (handwritten object rectangular areadisplay 503 in FIG. 18) counterclockwise according to the angularinformation acquired in step S60, and transmits the rotated rectangulararea display data to the display part 22 to be displayed thereby.

In step S64, the handwritten input display control part 23 transmits thestart of the selectable candidate display deletion timer to thecandidate display timer control part 24 in order to delete the selectedcandidate display data after a predetermined time elapses from the timewhen the selectable candidate display data are displayed. The candidatedisplay timer control part 24 starts the selectable candidate displaydeletion timer in response to receiving the start of the selectablecandidate display deletion timer.

Steps S65 through S70 are performed when the user deletes the selectablecandidate display displayed on the display part 22, or when the changeof the handwritten object occurs (that is, the stroke of the handwrittenobject is added, deleted, moved, deformed, or segmented), or when thecandidate is not selected before the time out, after the selectablecandidate delete timer is started.

Further, steps S65 through S67 illustrated in FIG. 38 are performed whenthe candidate display is deleted or the change in the handwritten objectoccurs.

In step S65, the handwritten input part 21 transmits the occurrence ofthe selectable candidate display deletion or the change in thehandwritten object to the handwritten input display control part 23.

In step S66, the handwritten input display control part 23 transmits thestop of the selectable candidate deletion timer. The candidate displaytimer control part 24 stops the selectable candidate deletion timer inresponse to receiving the stop of the selectable candidate deletiontimer, because an operation is performed on the handwritten objectwithin a predetermined time, and the selectable candidate deletion timeris no longer required.

In step S67, the handwritten input display control part 23 stores theposition information of the operation guide 500 in the handwritingrecognition control part 26 to be used for the gesture determination inthe gesture handwriting recognition of step S19. The positioninformation may be the coordinates of the upper left corner and thelower right corner, or similar coordinates, for example. Hence, thehandwriting recognition control part 26 can determine whether thestraight line used for inputting the angular information is inside theoperation guide 500.

In step S69, the handwritten input display control part 23 transmits thedeletion request for the selectable candidate display data to thedisplay part 22, to delete the selectable candidate display.

In step S70, the handwritten input display control part 23 transmits thedeletion request for the rectangular area display data of thehandwritten object and the selected object to the display part 22, todelete the rectangular area display. Accordingly, if the display of theoperation command candidates is deleted under conditions other than theselection of the operation command candidate, the display of thehandwritten object is maintained as is.

On the other hand, in step S68, if no deletion of the selectablecandidate display nor the change in the handwritten object occurs afterthe selectable candidate deletion timer is started (if the user does notperform the pen operation), the candidate display timer control part 24transmits the time out to the handwritten input display control part 23.

Similarly, after the time out of the selectable candidate displaydeletion timer, the handwritten input display control part 23 performssteps S69 and S70, because the display part 22 may delete the selectablecandidate display data, and rectangular area display data of thehandwritten object and the selected object, after the lapse of thepredetermined time.

If the user selects the selectable candidate after the selectablecandidate delete timer is started, steps S71 through S89 illustrated inFIG. 39 are performed.

In step S71, if the user selects the selectable candidate after theselectable candidate deletion timer is started, the handwritten inputpart 21 transmits the selection of the character string candidate or theoperation command candidate to the handwritten input display controlpart 23.

In step S71-2, the handwritten input display control part 23 transmitsthe stop of the selectable candidate display deletion timer to thecandidate display timer control part 24. The candidate display timercontrol part 24 stops the selectable candidate display deletion timer inresponse to receiving the stop of the selectable candidate displaydeletion timer.

Next, in step S72, the handwritten input display control part 23transmits a stored data clear to the handwriting recognition controlpart 26.

In step S73, the handwriting recognition control part 26 transmits thestored data clear to the character string conversion control part 28.

In step S74, the handwriting recognition control part 26 transmits thestored data clear to the predictive conversion control part 30.

In step S75, the handwriting recognition control part 26 transmits thestored data clear to the operation command recognition control part 32.

The handwriting recognition control part 26, the character stringconversion control part 28, the predictive conversion control part 30,and the operation command recognition control part 32 respectively clearthe data related to the character string candidates and the operationcommand candidates stored up to a point in time immediately beforereceiving the stored data clear.

Next, in step S76, the handwritten input display control part 23transmits the deletion of the selectable candidate display data to thedisplay part 22, to delete the selectable candidate display.

In step S77, the handwritten input display control part 23 transmits thedeletion of the rectangular area display data of the handwritten objectand the selected object to the display part 22, to delete therectangular area display.

In step S78, the handwritten input display control part 23 deletes thedisplay by transmitting the deletion of the handwritten object displaydata, and the deletion of the pen coordinate complement display datatransmitted in step S6, to the display part 22, to delete thehandwritten object display and the pen coordinate complement display.The handwritten object display and the pen coordinate complement displaymay be deleted, because the character string candidate or the operationcommand candidate is selected, thereby eliminating the need for thehandwritten object, or the like.

In step S79, the handwritten input display control part 23 transmits thedeletion of the handwritten object to the handwritten input storage part25.

If the character string candidate or the image candidate is selected bythe user, steps S80 through S82 are performed.

In step S80, when the character string candidate is selected, thehandwritten input display control part 23 transmits the addition of thecharacter string object to the handwritten input storage part 25. Whenthe image candidate is selected, the handwritten input display controlpart 23 transmits the addition of the image object to the handwritteninput storage part 25.

In step S81, the handwritten input display control part 23 transmits theacquisition for the character string object font to the handwritteninput storage part 25. The handwritten input storage part 25 selects adefined font from an estimated character size of the handwritten object,and transmits the selected font to the handwritten input display controlpart 23. If image candidate is selected, no particular processing isrequired because the image data is already acquired.

Next, in step S82, the handwritten input display control part 23transmits the character string object display data, which is to bedisplayed at the same position as the handwritten object, to the displaypart 22 using the defined font received from the handwritten inputstorage part 25, so as to display the character string object displaydata. When the image candidate is selected, the handwritten inputdisplay control part 23 transmits the image object display data, whichis to be displayed at the same position as the handwritten object, tothe display part 22, so as to display the image object display data.

If the operation command candidate is selected, steps S83 through S88are performed. Furthermore, steps S83 through S85 are performed if theselected object is present.

In step S83, when the operation command candidate for the selectedobject is specified (when the selected object is present), thehandwritten input display control part 23 transmits the deletion of theselected object display data to the display part 22, and deletes theselected object display, in order for the handwritten input displaycontrol part 23 to delete the original selected object.

Next, in step S84, the handwritten input display control part 23transmits the operation command for the selected object to thehandwritten input storage part 25. The handwritten input storage part 25transmits the display data (display data after editing or decorating) ofthe newly selected object to the handwritten input display control part23.

Next, in step S85, the handwritten input display control part 23transmits the selected object display data to the display part 22, sothat the selected object after executing the operation command isredisplayed.

In step S86, when the character string “

” pronounced “tegaki sain touroku suru” (“register handwrittensignature” in English) of the operation command definition data 713, orthe character string made up of six Kanji and Hiragana characters “

” pronounced “settei henkou suru” (“change setting” in English) of theoperation command definition data 716, which is the operation command ofthe input and output system, is specified, the handwritten input displaycontrol part 23 adds the handwritten signature registration form 561 orthe user definition data change form 562 to the handwritten inputstorage part 25.

In step S87, when other operation commands of the input and outputsystem are selected, the handwritten input display control part 23executes the operation command character string (Command) of theoperation command definition data corresponding to the operation commandselected by the user.

When the operation command 512 for signing in is executed, thehandwritten input display control part 23 acquires the pen ID receivedby the handwritten input part 21 upon execution of the operation command512. The handwritten input display control part 23 identifies the userdefinition data including “SignatureId” acquired in step S61, andacquires “AccountId” from the user definition data. Further, thehandwritten input display control part 23 registers “AccountId” in thepen ID control data in correspondence with the pen ID. As a result, thepen 2500 and the user become linked to each other, and the displaydevice 2 can perform processes using user definition data.

When the operation command is executed after the user has signed in, thehandwritten input display control part 23 identifies the user definitiondata using “AccountId”, and sets the identified user definition datainto “%˜%” of the operation command to execute the same. This“AccountId” in the pen ID control data corresponds to the pen IDreceived by the handwritten input part 21 upon execution of theoperation command.

In step S88, as illustrated in FIG. 29, if the user presses the rotatingoperation button 5111 of the operation header 520, the handwritten inputdisplay control part 23 accepts the angular information according to thenumber of times the rotating operation button 5111 is pressed. Thehandwritten input display control part 23 stores the accepted angularinformation in the pen ID control data storage part 36 in correspondencewith the pen ID received from the pen 2500 upon pressing of the rotatingoperation button 5111.

In step S89, for the next handwritten object, the handwritten inputdisplay control part 23 transmits the start of the handwritten object tothe handwritten input storage part 25. The handwritten input storagepart 25 secures the handwritten object region. Thereafter, the processesof steps S2 through S89 are repeated.

<Display Example of Image Candidates>

Hereinafter, a display example of the image candidates will bedescribed.

FIG. 40 illustrates an example of the operation guide 500 for displayingimage candidates 531 of pharmaceutical products. In FIG. 40, the userhandwrites a character string “

” pronounced “gasu” (“gas” in English), in order to display tablets withnames beginning with the character string “

” on the display device 2. The image transformation control part 42performs a search in the image transformation dictionary part 43 usingthe character string “

”, and displays two image candidates 531 with product names having aforward match with the character string “

”. In this example, the user intended to write a character string “

N⋅OD

” pronounced “gasuron enu oodijou (touroku shouhyou)” (“Gaslon N⋅ODTablets (registered trademark)” in English). However, because thecharacter string “

N⋅OD

” is similar to a character string “

” pronounced “gasukon jou (touroku shouhyou)” (“Gascon Tablets(registered trademark)” in English), there is a risk of erroneouslyselecting the latter similar tablet. However, because the distinctivepackages of the pharmaceutical products are displayed as the imagecandidates 531, the user can positively select the intended tablet beinghandwritten by the user. Such a display of the distinctive packages isalso useful in cases where the user can remember the package appearanceof the product but does not clearly remember the product name. In FIG.40, only the characters (product name) of the selected image candidate531 is displayed when the image candidate 531 of the operation guide 500is selected.

FIG. 41 illustrates an example of the operation guide for displayingimage candidates of photographs of people. In FIG. 41, the userhandwrites a Kanji character “

” pronounced “Sa”, in order to display the names of the people beginningwith the Kanji character “

” on the display device 2. The image transformation control part 42performs a search in the image transformation dictionary part 43 usingthe Kanji character “

”, and displays the four image candidates with the names having aforward match with the Kanji character “

”. Even if the names starting with the Kanji character “

” are similar, the image candidates 531 are photographs of the people,which enable the user to positively select the intended name of theperson being handwritten by the user. Such a display of the photographsof the people is also useful in cases where the user can remember theperson's face but does not clearly remember the person's name. In FIG.41, an image 1001 and the characters (person's name) of the selectedimage candidate 531 are displayed when the image candidate 531 of theoperation guide 500 is selected. The image 1001 may be displayed nearthe characters, such as above, below, on the left, or on the right ofthe characters (person's name). The image 1001 may be displayed atpositions other than near the characters (person's name).

FIG. 42 illustrates an example of the operation guide 500 for displayingmolecular structure models as the image candidates 531, similar to FIG.2. In FIG. 42, only the image 1001 (molecular structure model) isdisplayed when the user selects the image candidate 531. In other words,characters such as names are not displayed in this example. Such adisplay of only the image 1001 after the selection of image candidate531 is useful in cases where the structure of the image conveys themeaning of the information more effectively than the characters. Forexample, in a conference of experts, it may be possible to share notonly the name but also the functions by looking at the molecular model.In addition, when a teacher explains the shape or the like of an objectin a classroom or the like, it is easier to explain the structure of theobject when compared to using only the characters or names to explainthe structure.

As illustrated in FIG. 40 through FIG. 42, when the image candidate 531of the operation guide 500 is selected, the display device 2 can displayonly the image, the image and the name, or only the name. Because thepreferred display method depends on the user and the situation, the usercan set the display method from the operation header 520, or from a menuthat is displayed in response to the long press of the operation guide500.

<3DCG Image Candidates>

FIG. 43 through FIG. 45 illustrate examples of the operation guide 500that display the 3DCG of the molecular structure models as the imagecandidates. In other words, the image candidates are generated by the3DCG. The display method employed in FIG. 43 through FIG. 45 may besimilar to that employed in FIG. 19 through FIG. 21. In FIG. 43 throughFIG. 45, because the image candidates 531 displayed by the operationguide 500 are the 3DCG, the user can rotate the image candidates 531 up,down, left, and right using the pen 2500 or the like. Hence, the usercan confirm the structure that is not visible on the display at firstbut becomes visible after being rotated on the display, for example,before selecting the image candidate 531.

Because the display of the 3DCG requires a special viewer software, theimage candidate 531 may be displayed in a window separate from theoperation guide 500. The selected image candidate 531 can be displayedin 3DCG even after the image candidates 531 is selected. Such a displayis useful when explaining the structure of the product in a presentationor the like, and when the teacher teaches the structure of the object tostudents.

<Display of Related Image Candidates>

The image candidate 531 that is displayed is not necessarily the imagecandidate having the forward match with the recognized character stringof the handwritten data input by the user. For example, the displaydevice 2 may display the image candidate 531 that is related to theimage candidate having the forward match with the recognized characterstring of the handwritten data input by the user.

FIG. 46 illustrates an example of the operation guide 500 for displayingimage candidates 531 of the pharmaceutical products. In FIG. 46, theuser handwrites Katakana characters “

” pronounced “Depa”, in order to display drugs having similar utility oreffect as the drug having the name beginning with the character string “

” on the display device 2. The image transformation control part 42performs a search in the image transformation dictionary part 43 usingthe character string “

”, and identifies the image candidate 531 with product name “

” pronounced “Depasu” (“Depas (registered trademark” in English) havinga forward match with the character string “

”. In addition, the image transformation control part 42 identifies theimage candidates 531 having the name “

” pronounced “saireisu” (“Silece or Flunitrazepam (registeredtrademark)” in English) and the name “

” pronounced “renndorumin” (“Lendormin (registered trademark)” inEnglish) related to the product name “

” pronounced “Depasu”. The user can view the image candidates and selectthe intended or appropriate drug. For example, the user can select thedrug even when the user does not fully remember the name of the drug,and can select the most appropriate drug from among the drugs havingsimilar utility or effect. Further, the image candidates 531 havingnames beginning with different characters can be displayed.

In order to display the related image candidates 531 described above,the images of the image transformation dictionary part 43 may be relatedas illustrated in FIG. 47. In FIG. 47, the image transformationdictionary part 43 includes the item of the group ID, and the group IDis set with respect to the image. The same group ID is assigned to theimages belonging to the same group. The group ID having the same valuemay be assigned when there is similarity from a predetermined viewpoint,such as the shape, the properties, the usages, or the like. Accordingly,the related image candidates 531 can be displayed, as illustrated inFIG. 46.

<Search By Handwritten Data>

This embodiment describes the case where the search is performed in theimage transformation dictionary part 43 using the character stringconverted from the handwritten data. However, the image transformationcontrol part 42 may perform the search in the image transformationdictionary part 43 directly using the handwritten data. In this case,the handwritten data may be related to the images of the imagetransformation dictionary part 43, as illustrated in FIG. 48. In FIG.48, the image transformation dictionary part 43 includes the item of thehandwritten data, and handwritten data of the names correspond to theimages. The image transformation control part 42 performs the search inthe image transformation dictionary part 43 using the handwritten datain real time, from the beginning of the handwritten data. For example, adesigner may segment the handwritten data of the image transformationdictionary part 43 in units of characters beforehand. The imagetransformation control part 42 can determine a search range of thehandwritten data of the image transformation dictionary part 43according to the ratio of the horizontal distance and the verticaldistance used for detecting the estimated character size. In otherwords, the units of characters can be 1 character if the ratio of thehorizontal distance and the vertical distance is 1:1, and can be 2 ifthe ratio of the horizontal distance and the vertical distance is 1:2,and so on.

In this case, the search can be performed by pattern matching, forexample. Further, because the handwritten data has a characteristicfeature for each individual user, the image transformation dictionarypart 43 may register the handwritten data of different users.

<Notable Advantageous Features>

As described above, because the display device 2 according to thisembodiment displays the images of the selectable candidates, it becomeseasier for the user to select the intended person or object, and reducethe possibility of presenting incorrect information. In addition,because the selected image can also be displayed, the information can bepresented in a visually understandable manner.

The user may handwrite without distinguishing between inputtingcharacters or the like, and inputting handwritten signature, and theuser may call and make access to the various operation commands and theoperation command 512 for signing in, without distinguishing betweenvarious operation commands and the operation commands 512.

Further, the display device 2 according to this embodiment does notrequire the selection of the operation menu and the selection of theoperation from the list of buttons, and can input the operation commandsimilar to handwriting the characters or the like. Because the operationcommand and the selectable candidate 530 are displayed simultaneously inthe operation guide, the user can use the display device 2 withoutdistinguishing between the input of the characters or the like and theselection of the operation command. The user can handwrite thehandwritten object or enclose the decided object with the enclosureline, to display arbitrary operation command candidates. Accordingly, itis possible to call and make access to an arbitrary function (such asthe editing function, the input and output function, the pen function,or the like) from the handwritten state. Consequently, step-by-stepoperations are no longer required to call and make access to a desiredfunction by pressing the menu button, thereby reducing the operatingprocedure from the state where the user is handwriting to the callingand making access to the arbitrary function.

Second Embodiment

In this embodiment, the display device 2 converts English handwrittendata into the image candidates. The configuration of the display device2 in this embodiment is the same as that of the first embodiment, exceptthat the conversion dictionary and the operation command definition datacorrespond to the English language. For this reason, the features of thedisplay device 2 that are different from those of the first embodimentwill be described, based on the conversion of the handwritten data intoEnglish (hereinafter, referred to as “English conversion”) will bedescribed.

FIG. 49 is a diagram illustrating an example of a plurality of imagecandidates displayed by the display device 2 in the case of the Englishconversion. In the description of FIG. 49, only the differences fromFIG. 2 will mainly be described.

(1) The user handwrites a character “a” to input the term “Asparticacid”.(2) The display device 2 displays the image candidates 531 starting withthe character “a”. In other words, even while the user is handwriting,the image candidates 531, such as graphics, illustrations, photographs,or the like matching the character string that is being handwritten, aredisplayed. In FIG. 49, the molecular structure models of ascorbic acid,aspartic acid, acetylene, acetaldehyde, and alanine are displayed.(3) When the user selects one of the image candidates 531, the displaydevice 2 displays the selected image together with the charactersrepresenting the selected image. The display device 2 may display onlythe characters or only the images.

Accordingly, even in the case of the English conversion, because thedisplay device 2 according to this embodiment displays the images of theselectable candidates, it becomes easier for the user to select theintended person or object, and reduce the possibility of presentingincorrect information. In addition, because the selected image can alsobe displayed, the information can be presented in a visuallyunderstandable manner.

<Example of Dictionary Data>

The dictionary data in the case of the English conversion will bedescribed, with reference to FIG. 50 through FIG. 52. In the descriptionof FIG. 50 through FIG. 52, the differences from FIG. 8 through FIG. 10will mainly be described. FIG. 50 illustrates an example of thedictionary data of the handwriting recognition dictionary part 27 usedfor the English conversion. The dictionary data of the handwritingrecognition dictionary part 27 illustrated in FIG. 50 indicates that thehandwritten character “a (state of the stroke data)” has a 0.90probability of being converted into the character “a”, and a 0.10probability of being converted into a character “o”.

FIG. 51 illustrates an example of the dictionary data of the characterstring conversion dictionary part 29 used for the English conversion. Inthe dictionary data of the character string conversion dictionary part29 illustrated in FIG. 51, the character “a” has a 0.55 probability ofbeing converted into the character string “ab”, and has a 0.45probability of being converted into the character string “AI”. Similarprobabilities apply to other character strings before conversion.

FIG. 52 illustrates an example of the dictionary data of the predictiveconversion dictionary part 31 used for the English conversion. In thedictionary data of the predictive conversion dictionary part 31illustrated in FIG. 52, the character string “agenda” has a 0.55probability of being converted into the character string “agenda list”,and has a 0.30 probability of being converted into the character string“agenda template”. Similar probabilities apply to other character andcharacter strings before conversion.

The dictionary data has no language dependency, and any character orcharacter string may be registered before and after conversion.

FIG. 53 illustrates an example of the image dictionary data of the imagetransformation dictionary part 43 used for the English conversion. Inthe description of FIG. 53, the differences from FIG. 11 will mainly bedescribed. In the case of English conversion, the English name of theimage is registered in the index. Accordingly, if the user handwrites inEnglish, the display device 2 can display images corresponding to theEnglish language.

<Example of Operation Command Definition Data>

FIG. 54A and FIG. 54B illustrate an example of operation commanddefinition data when no selected object is present. In the descriptionof FIG. 54A and FIG. 54B, the differences from FIG. 12A will mainly bedescribed. The contents of each of the operation commands are the sameas in FIG. 12A, but English expressions are made to correspond to theoperation command name (Name) and the character string (String).Accordingly, the user can handwrite the operation command in English,and select the operations command in English.

FIG. 55 illustrates an example of the operation command definition datawhen the selected object is present. In the description of FIG. 55, thedifferences from FIG. 13 will mainly be described. The contents of eachof the operation commands are the same as in FIG. 13, but Englishexpressions are made to correspond to “Name”. Accordingly, the user canselect the operation command in English.

FIG. 56 illustrates an example of the user definition data. In thedescription of FIG. 56, the differences from FIG. 14 will mainly bedescribed. The contents of user definition data are the same as in FIG.14, but “AccountUsername” is “Bob” in this example. In addition, Englishexpressions are made to correspond to “Name”. Accordingly, the user canselect the user definition data in English.

The handwritten signature data stored by the handwritten signature datastorage part 39 may be similar to those illustrated in FIG. 15, and thehandwritten input storage data may be similar to those illustrated inFIG. 16. The pen ID control data may be similar to those illustrated inFIG. 17A and FIG. 17B.

<Display Example of Selectable Candidates>

FIG. 57 illustrates an example of the operation guide 500, and theselectable candidate 530 displayed by the operation guide 500, in thecase of English conversion. In the description of FIG. 57, thedifferences from FIG. 18 will mainly be described. In FIG. 57, the userhandwrites the character “a” as the handwritten object 504. Theoperation command candidate 510, the handwriting recognition characterstring candidate 506, the converted character string candidate 507, andthe character string/predictive conversion candidate 508 are displayed,based on the character “a”. Accordingly, the display is similar to thatof FIG. 18, except that the Japanese expressions in FIG. 18 areconverted into English in FIG. 57. The button 501 is labeled “P-C” whichis an abbreviation for “Predictive Conversion”.

The operation command candidate 510 include the operation commanddefinition data 701 and 702 having “agenda” in “string” of the operationcommand definition data illustrated in FIG. 54A, for example.

Accordingly, the user can similarly cause the display of the operationguide 500, also in the case of the English conversion.

FIG. 58 through FIG. 60 illustrate transitions in which image candidates531 are gradually narrowed down according to the user's handwriting. Inthe description of FIG. 58 through FIG. 60, the differences from FIG. 19through FIG. 21 will mainly be described. Similar to the case of theJapanese language conversion, the handwriting recognition control part26 successively converts the handwritten data from the beginning into acharacter or character string. The image transformation control part 42performs the search in the image transformation dictionary part 43 bythe forward match, while increasing the number of converted charactersevery time the character is converted. The display part 22 displays anumber of image candidates, which decreases as the number of charactersused for the search increases.

In FIG. 58, the user handwrites the character “a” as the handwrittenobject 504. When the character “a” is recognized, and “a” is obtained asthe handwriting recognition character string, the handwritten inputdisplay control part 23 transmits the character “a” to the imagetransformation control part 42. The image transformation control part 42performs the search in the image transformation dictionary part 43illustrated in FIG. 53, and transmits the image candidate correspondingto the index field including the name having a forward match with thecharacter “a” to the image transformation dictionary part 43. As aresult, in FIG. 58, five image candidates 531 having the names beginningwith the character “a” are displayed in the operation guide 500.

Next, FIG. 59 illustrates the image candidate 531 displayed when theuser handwrites the character “a”, and thereafter handwrites thecharacter “s” without selecting the image candidate 531. In this case,because the user handwrites the character string “as”, the handwritteninput display control part 23 transmits the character string “as” to theimage transformation control part 42. The image transformation controlpart 42 performs the search in the image transformation dictionary part43 illustrated in FIG. 53, and transmits the image candidatecorresponding to the index field including the name having a forwardmatch with the character string “as” to the handwritten input displaycontrol part 23. As a result, in FIG. 59, two image candidates 531having the names (“aspartic acid” and “ascorbic acid”) beginning withthe character string “as” are displayed in the operation guide 500.

As described above, the image transformation control part performs thesearch in the image transformation dictionary part by the forward match,while increasing the number of converted characters every time thecharacter is converted.

Next, FIG. 60 illustrates the image candidate 531 that is displayed whenthe user handwrites entire the character string “Aspartic”. Thehandwritten input display control part 23 transmits the character string“Aspartic” to the image transformation control part 42. The imagetransformation control part 42 performs the search in the imagetransformation dictionary part 43 illustrated in FIG. 53, and transmitsthe image candidate corresponding to the index field including the namehaving a forward match with the character string “Aspartic” to thehandwritten input display control part 23. As a result, in FIG. 60, theimage candidate 531 having the name “Aspartic acid” is displayed in theoperation guide 500.

FIG. 61 through FIG. 63 illustrate an example of display of the imagecandidate 531, and the operation guide 500 for displaying the imagecandidate name 532. FIG. 61 corresponds to FIG. 58, FIG. 62 correspondsto FIG. 59, and FIG. 63 corresponds to FIG. 60, and the display methodillustrated in FIG. 61 through FIG. 63 may be the same as thatillustrated in FIG. 58 through FIG. 60. In the case of FIG. 61 throughFIG. 63, the image transformation control part 42 also receives the nameof the index field together with the image from the image transformationdictionary part 43, and transmits the received name and image to thehandwritten input display control part 23. Accordingly, the operationguide 500 can display the name of the image candidate.

<Example of Specifying Selected Object>

FIG. 64A and FIG. 64B illustrate an example of a diagram illustrating aspecification example of a selected object in the case of an Englishconversion. In the description of FIG. 64A and FIG. 64B, the differencesfrom FIG. 25A through FIG. 25D will mainly be described.

FIG. 64A illustrates an example in which two decided objects 13 a 2 and13 b 2 written horizontally are specified by the user using the stridingline (handwritten object 11 a 2). In this example, the length H1 of theshorter side and the length W1 of the longer side of the handwrittenobject rectangular region 12 a 2 satisfy the striding line determinationcondition 406, and the overlap rate of the handwritten objectrectangular region 12 a 2 with respect to the decided objects 13 a 2 and13 b 2 satisfies the striding line determination condition 406. For thisreason, the decided objects 13 a 2 and 13 b 2 of both “agenda” and “ag”are specified as the selected objects.

FIG. 64B illustrates an example in which the decided object 13 c 2written horizontally is specified by the user using the enclosure line(handwritten object 11 b 2). In this example, only the decided object 13c 2 “agenda”, which satisfies the enclosure line determination condition407, is specified as the selected object.

Accordingly, in the case of English conversion, the user can similarlyselect the decided object.

<Display Example of Operating Command Candidates>

FIG. 65A and FIG. 65B illustrate a display example of the operationcommand candidates based on the operation command definition data in thepresence of the handwritten object illustrated in FIG. 64A and FIG. 64B.In the description of FIG. 65A and FIG. 65B, the differences from FIG.26A and FIG. 26B will mainly be described.

FIG. 65A illustrates the operation command candidate for the editingsystem, and FIG. 65B illustrates the operation command candidate for thedecorating system. FIG. 65A illustrates an example in which the decidedobject is specified in the handwritten object 11 a 2 illustrated in FIG.64A. As illustrated in FIG. 65A and FIG. 65B, the main menu 550 includesthe operation command candidate displayed after the bullet character“>>” 511.

In FIG. 65A, the sub menu 560 illustrated in FIG. 65A is displayed bypressing the end-of-line character “>” 512 a of the first line. When theuser presses any of the operation command names by the pen, thehandwritten input display control part 23 executes the “Command” of theoperation command definition data corresponding to the operation commandname with respect to the selected object. In other words, “Delete” isexecuted when a “Delete” button 521 b is selected, “Move” is executedwhen a “Move” button 522 b is selected, “Rotate” is executed when a“Rotate” button 523 b is selected, and “Select” is executed when a“Select” button 524 b is selected.

When the user presses the “Delete” button 521 b with the pen, thehandwritten input display control part 23 deletes the decided objects 13a 2 and 13 b 2 “agenda” and “ag”. When the user presses the “Move”button 522 b with the pen, the handwritten input display control part 23accepts the movement of the decided objects 13 a 2 and 13 b 2 “agenda”and “ag”. When the user presses the “Rotate” button 523 b with the pen,the handwritten input display control part 23 rotates the decidedobjects 13 a 2 and 13 b 2 “agenda” and “ag” by a predetermined angle.When the user presses the “Select” button 524 b with the pen, thehandwritten input display control part 23 accepts the selection of thedecided objects 13 a 2 and 13 b 2 “agenda” and “ag”.

Character string candidates other than the operation command candidates,such as “-” 541 b, “-,” 542 b, “˜” 543 b, “→” 544 b, and “⇒” 545 b, arethe recognition results of the striding line (handwritten object 11 a2). Hence, if the user intends to input the character string and not theoperation command, the character string candidate can be selected.

In FIG. 65B, when the user presses the end-of-line character “>” 512 bof the second line, the sub menu 560 is displayed on the right sidethereof. Similar to FIG. 65A, FIG. 65B illustrates the example in whichboth the main menu 550 and the sub menu 560 are displayed. When “Thick”531 b is selected based on the operation command definition dataillustrated in FIG. 55, the handwritten input display control part 23executes “Thick” on the selected object to make the selected objectthick. When “Thin” 532 b is selected, the handwritten input displaycontrol part 23 executes “Thin” with respect to the selected object tomake the selected object thin. When “Large” 533 b is selected, thehandwritten input display control part 23 executes “Large” with respectto the selected object to make the selected object large. When “Small”534 b is selected, the handwritten input display control part 23executes “Small” with respect to the selected object to make theselected object small. When “Underline” 535 b is selected, thehandwritten input display control part 23 executes “Underline” withrespect to the selected object to underline the selected object.

When the user presses “Thick” 531 b with the pen, the handwritten inputdisplay control part 23 thickens the lines forming the decided objects13 a 2 and 13 b 2 “agenda” and “ag”. When the user presses “Thin” 532 bwith the pen, the handwritten input display control part 23 narrows thelines forming “agenda” and “ag”. When the user presses “Large” 533 bwith the pen, the handwritten input display control part 23 enlarges thecharacters. When the user presses “Small” 534 b with the pen, thehandwritten input display control part 23 reduces the characters. Whenthe user presses “Underline” 535 b with the pen, the handwritten inputdisplay control part 23 can add underlines to the characters.

Accordingly, the user can cause the operation commands to be displayedwhen the handwritten object is present, even in the case of the Englishconversion.

<Input Example of Angular Information>

Next, FIG. 66A through FIG. 66C are diagrams illustrating an example ofthe input method of angular information. In the case of the Englishconversion, the input method of the angular information is the same asthat of the Japanese conversion. In the description of FIG. 66A throughFIG. 66C, the differences from FIG. 28A through FIG. 28C are mainlydescribed.

FIG. 66A illustrates a state where the operation guide 500 is displayedbecause the user positioned in the 3 o'clock direction from the displaydevice 2 handwrites the character “a” in the state where the angularinformation of the pen ID control data is 0 degrees (initial value).Because the display device 2 recognizes the character “a” handwrittenfrom the 3 o'clock direction with the angular information maintained at0 degrees, the selectable candidate 530 that is different from theanticipated selectable candidate is displayed.

When inputting the angular information, the user handwrites the straightline from the top to bottom when viewed from the user. FIG. 66Billustrates an example of a straight line 5211 b. The angularinformation is the angle α formed in the counterclockwise directionbetween the straight line 5211 b and the 6 o'clock direction which isthe angular information of 0 degrees.

FIG. 66C illustrates the operation guide 500 immediately after theoperation illustrated in FIG. 66B is performed. Because 90 degrees isset as the angular information (“Angle”) in the pen ID control data, thehandwritten object (stroke data) is internally rotated clockwise by 90degrees to be subject to the handwriting recognition, and the operationguide 500 is rotated counterclockwise by 90 degrees and displayed. Theangular information may be input manually by the user from the menu.

Accordingly, the user can input the angular information also in the caseof English conversion.

<Registration Example of Handwritten Signature Data>

Next, FIG. 67A through FIG. 67C are diagrams illustrating a method ofregistering handwritten signature data. In the description of FIG. 67Athrough FIG. 67C, the differences from FIG. 30A through FIG. 30C aremainly described. First, FIG. 67A illustrates an example of theselectable candidate 530 displayed when the user handwrites thecharacter string “Sign”. The two operation commands 513 b and 514 b“Register handwritten signature” and “Hand sign out” based on theoperation command definition data 713 and 715, and the character stringcandidates “Sign”, “Signature session”, and “Signed”, which partialmatch the character string “Sign”, are displayed.

When the user presses “Register handwritten signature” with the pen2500, the handwritten signature registration form 561 illustrated inFIG. 67B is added to the handwritten input storage part 25, and isdisplayed on the operation screen 101. For example, the operation guide500 illustrated in FIG. 67A is deleted, and the handwritten signatureregistration form 561 is displayed at the same position as the operationguide 500.

The user inputs the text of the name in the name input field 561 a, thefirst-time handwritten signature in the signature input field 561 b, thesecond-time handwritten signature in the signature input field 561 c,and the third-time handwritten signature in the signature input field561 d, and the check mark or cancellation mark in the registrationconfirmation field 561 e.

Accordingly, when the user writes the handwritten signature registrationform 561 as directed, the handwritten signature registration form 561becomes the handwritten signature registration form 561 illustrated inFIG. 67C.

As described above, the handwritten input display control part 23 canaccept the handwritten input without distinguishing between thehandwritten input to the form and the handwritten input to other thanthe form. The user can register the handwritten signature data usingEnglish.

<Example of Handwritten Sign In>

FIG. 68 illustrates an example of the operation guide 500 that isdisplayed in the case of the English conversion, when the userhandwrites the character string “Bob”, which is the registeredhandwritten signature data. In the description of FIG. 68, thedifferences from FIG. 31 will mainly be described.

Because the character string “Bob” is registered in the handwrittensignature data storage part 39 as the handwritten signature data, thecharacter string “Bob” matches the registered handwritten signaturedata. Hence, the operation command 512 “Hand Sign in”, which is anabbreviation for “Handwritten sign in”, is displayed.

In addition, because the handwritten signature data matches theregistered handwritten signature data, “SignatureId” representing “Bob”is identified, and user definition data including “AccountId”corresponding to “SignatureId” is identified.

When the user selects the operation command 512 “Hand Sign in”,“AccountID” of “Bob” is added to the pen ID control data incorrespondence with the pen ID of the pen 2500 that is being used by theuser, so that the user definition data of the character string “Bob” isused when utilizing the operation command.

Accordingly, the user can sign in using English.

<Example of User Definition Data Changing Operation>

FIG. 69A and FIG. 69B are diagrams illustrating a method of changing theuser definition data in the case of the English conversion. In thedescription of FIG. 69A and FIG. 69B, the differences from FIG. 32A andFIG. 32B will mainly be described. FIG. 69A illustrates an example ofthe operation guide 500 that is displayed when the user handwrites thecharacter string “set”. The operation command definition data 716defines the character string “set” in “String”, and the operationcommand “Change settings” is displayed.

If the user selects “Change setting” with the pen 2500 that was used forthe handwritten sign in, “AccountId” corresponding to the pen ID of thispen 2500 is identified from the pen ID control data. Hence, the userdefinition data of the user who signed is identified, and the userdefinition data change form 562 illustrated in FIG. 69B is added to thehandwritten input storage part 25 and displayed on the operation screen101. Each of the items of the user definition data change form 562 issimilar to that in FIG. 32B.

Accordingly, the user can change the user definition data in the case ofEnglish conversion, similar to the Japanese conversion.

<Display Example of Image Candidates>

Hereinafter, a display example of the image candidates in the case ofEnglish conversion will be described.

FIG. 70 illustrates an example of the operation guide 500 for displayingimage candidates 531 of pharmaceutical products. In the description ofFIG. 70, the differences from FIG. 40 will mainly be described. In FIG.70, the user handwrites the character string “gas”, in order to displaytablets having names beginning with the character string “gas” on thedisplay device 2. The image transformation control part 42 performs thesearch in the image transformation dictionary part 43 using thecharacter string “gas”, and displays two image candidates 531 having aforward match with the character string “gas”. In this example, the userintended to display “Gaslon N⋅OD Tablets (registered trademark”.However, because “Gaslon N⋅OD Tablets (registered trademark)” is similarto “GASCON (registered trademark)”, there is a possibility oferroneously selecting “GASCON (registered trademark)”. Even in such acase, because the distinctive packages of the pharmaceutical productsare displayed as the image candidates 531, the user can positivelyselect the intended tablet being handwritten by the user.

Such a display of the distinctive packages is also useful in cases wherethe user can remember the package appearance of the product but does notclearly remember the product name. In FIG. 70, only the characters(product name) of the selected image candidate 531 is displayed when theimage candidate 531 of the operation guide 500 is selected.

FIG. 71 illustrates an example of the operation guide for displaying theimage candidates of photographs of people. In the description of FIG.71, the differences from FIG. 41 will mainly be described. In FIG. 71,the user handwrites the character “B”, in order to display the names ofthe people beginning with the character “B” on the display device 2. Theimage transformation control part 42 performs the search in the imagetransformation dictionary part 43 using the character “B”, and displaysthe four image candidates with the names having a forward match with thecharacter “B”. Even if the names starting with the character “B” aresimilar, the image candidates 531 are photographs of the people, whichenable the user to positively select the intended name of the personbeing handwritten by the user. Such a display of the photographs of thepeople is also useful in cases where the user can remember the person'sface but does not clearly remember the person's name. In FIG. 71, theimage 1001 and the characters (person's name) of the selected imagecandidate 531 are displayed when the image candidate 531 of theoperation guide 500 is selected. The image 1001 may be displayed nearthe characters, such as above, below, on the left, or on the right ofthe characters (person's name). The image 1001 may be displayed atpositions other than near the characters (person's name).

FIG. 72 illustrates an example of the operation guide 500 for displayingmolecular structure models as the image candidates 531, similar to FIG.2. In the description of FIG. 72, the differences from FIG. 42 willmainly be described. In FIG. 72, only the image 1001 (molecularstructure model) is displayed when the user selects the image candidate531. In other words, characters such as the names are not displayed inthis example. Such a display of only the image 1001 after the selectionof image candidate 531 is useful in cases where the structure of theimage conveys the meaning of the information more effectively than thecharacters. For example, in a conference of experts, it may be possibleto share not only the name but also the functions by looking at themolecular model. In addition, when a teacher explains the shape or thelike of an object in a classroom or the like, it is easier to explainthe structure of the object when compared to using only the charactersor names to explain the structure.

As illustrated in FIG. 70 through FIG. 72, when the image candidate 531of the operation guide 500 is selected, the display device 2 can displayonly the image, the image and the name, or only the name, also in thecase of the English conversion. Because the preferred display methoddepends on the user and the situation, the user can set the displaymethod from the operation header 520, or from a menu that is displayedin response to the long press of the operation guide 500.

<3DCG Image Candidates>

FIG. 73 through FIG. 75 illustrate examples of the operation guide 500that display the 3DCG of the molecular structure models as the imagecandidates, in the case of the English conversion. In the description ofFIG. 73 through FIG. 75, the differences from FIG. 43 through FIG. 45will mainly be described. In FIG. 73 through FIG. 75, the imagecandidates are generated by the 3DCG. The display method employed inFIG. 73 through FIG. 75 may be similar to that employed in FIG. 19through FIG. 21. In FIG. 73 through FIG. 75, because the imagecandidates 531 displayed by the operation guide 500 are the 3DCG, theuser can rotate the image candidates 531 up, down, left, and right usingthe pen 2500 or the like. Hence, the user can confirm the structure thatis not visible on the display at first but becomes visible after beingrotated on the display, for example, before selecting the imagecandidate 531.

<Display of Related Image Candidates>

FIG. 76 illustrates an example of the operation guide 500 for displayingimage candidates 531 of the pharmaceutical products in the case of theEnglish conversion. In the description of FIG. 76, the differences fromFIG. 46 will mainly be described. In FIG. 76, the user handwrites acharacter string “DE”, in order to display drugs having similar utilityor effect as the drug having the name beginning with the characterstring “DE” on the display device 2. The image transformation controlpart 42 performs the search in the image transformation dictionary part43 using the character string “DE”, and identifies the image candidate531 with product name “DEPAS (registered trademark)” having a forwardmatch with the character string “DE”. In addition, the imagetransformation control part 42 identifies the image candidates 531having the name “Flunitrazepam (registered trademark)” and the name“Lendormin (registered trademark)” related to the product name “DEPAS(registered trademark)”. The user can view the image candidates 531 andselect the intended or appropriate drug. For example, the user canselect the drug even when the user does not fully remember the name ofthe drug, and can select the most appropriate drug from among the drugshaving similar utility or effect. Further, the image candidates 531having names beginning with different characters can be displayed.

In order to display the related image candidates 531 described above,the images of the image transformation dictionary part 43 may be relatedas illustrated in FIG. 77. In FIG. 77, the image transformationdictionary part 43 includes the item of the group ID, and the group IDis set with respect to the image. The same group ID is assigned to theimages belonging to the same group. The group ID having the same valuemay be assigned when there is similarity from a predetermined viewpoint,such as the shape, the properties, the usages, or the like. Accordingly,the related image candidates 531 can be displayed, as illustrated inFIG. 76.

<Search By Handwritten Data>

The image transformation control part 42 may perform the search in theimage transformation dictionary part 43 directly using the handwrittendata, also in the case of the English conversion. In this case, thehandwritten data may be related to the images of the imagetransformation dictionary part 43, as illustrated in FIG. 78. In thedescription of FIG. 78, the differences from FIG. 48 will mainly bedescribed. In FIG. 78, the image transformation dictionary part 43includes the item of the handwritten data, and handwritten data of thenames correspond to the images.

The operating procedure may be similar to that of the first embodimentillustrated in FIG. 33 through FIG. 39.

Third Embodiment

In the first embodiment, it is assumed that the display device 2includes a large touchscreen panel. However, the display device 2 is notlimited to the touchscreen panel. In this embodiment, the display deviceis a projector type display device.

<First Example of Display Device Configuration>

FIG. 79 illustrates another configuration example of a display device.In FIG. 79, a projector 432 is provided above a conventional whiteboard434. This projector 432 corresponds to the display device. Theconventional whiteboard 434 is not a flat panel display integral withthe touchscreen panel, but is a whiteboard on which the user writesdirectly with a marker pen. The whiteboard may be a blackboard, andsimply needs to have a sufficiently large flat surface that enablesimages to be projected thereon.

The projector 432 includes an ultra short focus optical system, so thatlow-distortion images can be projected onto the whiteboard 434 from adistance of approximately 10 cm. The images may be transmitted from a PCor the like having a wireless or wired connection to the projector 432.Alternatively, the images may be stored in the projector 432.

The user handwrites on the whiteboard 434 using a dedicated electronicpen 2501. The electronic pen 2501 has a light emitting part at a tipportion, for example, and the light emitting part turns on when the userpresses the pen tip against the whiteboard 434 for handwriting. Thewavelength of light emitted from the light emitting part isnear-infrared or infrared, and is invisible to the user's eyes. Theprojector 432 includes a camera that captures the light emitting partand analyzes the captured image to determine the direction of theelectronic pen 2501. The electronic pen 2501 emits a sound wave togetherwith the light, and the projector 432 calculates a distance from theelectronic pen 2501 according to the arrival time of the sound wave. Theprojector 432 can identify the position of the electronic pen 2501 fromthe determined direction and the calculated distance. A stroke is drawn(projected) at the position of the electronic pen 2501.

Because the projector 432 projects a menu 430, when the user presses abutton with the electronic pen 2501, the projector 432 identifiespressed button from the position of the electronic pen 2501 and anon-signal of a switch. For example, when a store button 431 is pressed,a stroke (a set of coordinates) handwritten by the user is stored in theprojector 432. The projector 432 stores handwritten information in apredetermined server 433, a USB memory 2600, or the like. Thehandwritten information may be stored in units of pages. The coordinatesare stored instead of the image data, to facilitate reediting thereof bythe user. In this embodiment, however, the display of the menu 430 isnot essential, because the operation commands can be called and accessedby the handwriting.

<Second Example of Display Device Configuration>

FIG. 80 is a diagram illustrating another configuration example of thedisplay device 2. In the example illustrated in FIG. 80, the displaydevice 2 includes a terminal device 600, an image projector device 700A,and a pen operation detecting device 810.

The terminal device 600 is wire-connected to the image projector device700A and the pen operation detecting device 810. The image projectordevice 700A projects the image data input from the terminal device 600onto a screen 800.

The pen operation detecting device 810 communicates with an electronicpen 820, and detects the operation (or motion) of the electronic pen 820in a vicinity of the screen 800. More particularly, the electronic pen820 detects coordinate information indicating a point on the screen 800indicated (or pointed)) by the electronic pen 820, and transmits thecoordinate information to the terminal device 600.

The terminal device 600 generates image data of a stroke image input bythe electronic pen 820, based on the coordinate information receivedfrom the pen operation detecting device 810. The terminal device 600control the image projector device 700A to draw the stroke image on thescreen 800.

In addition, the terminal device 600 generates superimposed image datarepresenting a superimposed image composed of a background imageprojected by the image projector device 700A and the stroke image inputby the electronic pen 820.

<Third Example of Display Device Configuration>

FIG. 81 is a diagram illustrating another configuration example of thedisplay device. In the example illustrated in FIG. 81, the displaydevice 2 includes a terminal device 600, a display 800A, and a penoperation detecting device 810A.

The pen operation detecting device 810A is arranged near the display800A, and detects coordinate information indicating a point on thedisplay 800A indicated (or pointed) by an electronic pen 820A, andtransmits the coordinate information to the terminal device 600. In theexample illustrated in FIG. 81, the electronic pen 820A may be chargedby the terminal device 600 via a USB connector.

The terminal device 600 generates image data of a stroke image input bythe electronic pen 820A, and displays the image data on the display 800Abased on the coordinate information received from the pen operationdetecting device 810A.

<Fourth Example of Display Device Configuration>

FIG. 82 is a diagram illustrating another configuration example of thedisplay device. In the example illustrated in FIG. 82, the displaydevice 2 includes a terminal device 600 and an image projector device700A.

The terminal device 600 performs a wireless communication with anelectronic pen 820B, via Bluetooth (registered trademark) or the like,and receives coordinate information of a point on the screen 800indicated (or pointed) by the electronic pen 820B. The terminal device600 generates the image data of the stroke image input by the electronicpen 820B, based on the received coordinate information, and controls theimage projector device 700A to project the stroke image.

The terminal device 600 generates superimposed image data representing asuperimposed image composed of a background image projected by the imageprojector device 700A and the stroke image input by the electronic pen820B.

As described above, each of the above described embodiments can beapplied to various system configurations.

<Other Applications>

While preferred embodiments of the present invention are described abovewith reference to examples, various variations, modifications, andsubstitutions may be made thereto without departing from the spirit andscope of the present disclosure.

For example, the display method of the embodiments is suitablyapplicable to an information processing apparatus having a touchscreenpanel. Devices having the same function as the display device are alsoreferred to as electronic chalkboards, electronic whiteboards,electronic information boards, interactive boards, or the like. Theinformation processing apparatus having the touchscreen panel may be anoutput device such as a projector (PJ), a digital signage, or the like,a Head Up Display (HUD) device, an industrial machine, an imagingdevice, a sound collector, a medical device, a network home appliance, alap-top Personal Computer (PC), a cellular phone, a smartphone, a tabletterminal, a game device, a Personal Digital Assistant (PDA), a digitalcamera, a wearable PC, a desktop PC, or the like, for example.

According to the embodiments, a part of the processes performed by thedisplay device 2 may be performed by the server. For example, thedisplay device may transmit stroke information to the server, acquireinformation to be displayed in the operation guide 500 from the server,and display the acquired information in the operation guide 500. In thiscase, the display device 2 may form a display system.

In the embodiments, the coordinates of the pen tip are detected by themethod of detecting the coordinates of the pen tip by the touchscreenpanel. However, the display device 2 may detect the coordinates of thepen tip using ultrasonic waves. In addition, the pen may emit ultrasonicwaves together with light, and the display device 2 may calculate thedistance from the pen according to the arrival time of the ultrasonicwaves. The display device 2 can locate the position of the pen from thedetected direction and the calculated distance. The projector can draws(projects) the pen's trajectory as a stroke.

In the embodiments, the operation command candidates for the editingsystem and the decorating system are displayed when the selected objectis present, however, such operation command candidates may be displayedsimultaneously with other operation command candidates.

Further, the configuration example such as that of FIG. 6A is dividedaccording to the main function, in order to facilitate understanding ofthe processes of the display device 2. The present disclosure is notlimited by the method of dividing the processes in units or by names.The processes of the display device 2 can further be divided intosmaller processing units depending on the processing contents, forexample. Alternatively, one processing unit may be split to include moreprocesses.

According to each of the embodiments described above, it is to provide adisplay device for displaying information other than characters, as theselectable candidates.

Although the embodiments and the examples are numbered with, forexample, “first,” “second,” “third,” etc., the ordinal numbers do notimply priorities of the embodiments and the examples.

Numerous additional modifications and variations are possible in lightof the above teachings. It is therefore to be understood that, withinthe scope of the appended claims, the disclosure of this patentspecification may be practiced otherwise than as specifically describedherein.

As can be appreciated by those skilled in the computer arts, thisinvention may be implemented as convenient using a conventionalgeneral-purpose digital computer programmed according to the teachingsof the present specification. Appropriate software coding can readily beprepared by skilled programmers based on the teachings of the presentdisclosure, as will be apparent to those skilled in the software arts.The present invention may also be implemented by the preparation ofASICs or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in therelevant art.

Each of the functions of the described embodiments may be implemented byone or more processing circuits. A processing circuit may encompass aprogrammed processor. A processing circuit may also encompass devicessuch as an application specific integrated circuit (ASIC) andconventional circuit components arranged to perform the recitedfunctions.

The processing circuitry is implemented as at least a portion of amicroprocessor. The processing circuitry may be implemented using one ormore circuits, one or more microprocessors, microcontrollers, ASICs,dedicated hardware, DSPs, microcomputers, central processing units,FPGAs, programmable logic devices, state machines, super computers, orany combination thereof. Also, the processing circuitry may encompassone or more software modules executable within one or more processingcircuits. The processing circuitry may further encompass a memoryconfigured to store instructions and/or code that causes the processingcircuitry to execute functions.

If embodied in software, each block may represent a module, segment, orportion of code that comprises program instructions to implement thespecified logical function(s). The program instructions may be embodiedin the form of source code that comprises human-readable statementswritten in a programming language or machine code that comprisesnumerical instructions recognizable by a suitable execution system suchas a processor in a computer system or other system. The machine codemay be converted from the source code, or the like. If embodied inhardware, each block may represent a circuit or a number ofinterconnected circuits to implement the specified logical function(s).

What is claimed is:
 1. A display device comprising: a circuitryconfigured to perform a search for a plurality of image candidates in animage transformation dictionary part, based on handwritten data; and adisplay configured to display the plurality of image candidates obtainedby the search, wherein at least a portion of the plurality of imagecandidates displayed on the display represents a different person or anobject.
 2. The display device as claimed in claim 1, wherein thecircuitry is configured to recognize the handwritten data and convertthe recognized handwritten data into one or more characters, and performthe search based on the one or more converted characters, and thedisplay displays the plurality of image candidates obtained from theimage transformation dictionary part by the search.
 3. The displaydevice as claimed in claim 2, wherein the circuitry is configured tosuccessively convert the handwritten data from the beginning into theone or more characters, and performs the search in the imagetransformation dictionary part by a forward match while increasing thenumber of converted characters every time a character is converted, andthe display displays a number of image candidates which decreases as thenumber of converted characters used for the search increases.
 4. Thedisplay device as claimed in claim 1, wherein the display displays namesof the plurality of image candidates together with the plurality ofimage candidates.
 5. The display device as claimed in claim 1, whereinthe plurality of image candidates are 3-Dimensional Computer Graphics(3DCG) rotatable by 360 degrees.
 6. The display device as claimed inclaim 1, wherein the image transformation dictionary part is set withidentification information of a group to which the plurality of imagecandidates belong, and the display displays a first image candidateamong the plurality of image candidates obtained from the imagetransformation dictionary part by the search based on handwritten data,and at least a second image candidate among the plurality of imagecandidates having identification information of the same group as thefirst image candidate.
 7. The display device as claimed in claim 6,wherein image candidates having similar shapes, properties, orapplications, among the plurality of image candidates, haveidentification information of the same group.
 8. The display device asclaimed in claim 1, further comprising: a handwritten input partconfigured to accept a selection of one of the plurality of imagecandidates, wherein the display deletes all of the plurality of imagecandidates and the handwritten data, and displays the selected one ofthe plurality of image candidates accepted by the handwritten input partat the same position as the handwritten data.
 9. The display device asclaimed in claim 1, further comprising: a handwritten input partconfigured to accept a selection of one of the plurality of imagecandidates, wherein the display deletes all of the plurality of imagecandidates and the handwritten data, and displays a name of the selectedone of the plurality of image candidates accepted by the handwritteninput part at the same position as the handwritten data.
 10. The displaydevice as claimed in claim 1, further comprising: a handwritten inputpart configured to accept a selection of one of the plurality of imagecandidates, wherein the display deletes all of the plurality of imagecandidates and the handwritten data, displays a name of the selected oneof the plurality of image candidates accepted by the handwritten inputpart at the same position as the handwritten data, and displays theselected one of the plurality of image candidates in a periphery of thedisplayed name.
 11. A display method comprising: searching, by an imagetransformation control part, a plurality of image candidates in an imagetransformation dictionary part, based on handwritten data; anddisplaying, by a display, the plurality of image candidates obtained bythe search, wherein at least a portion of the plurality of imagecandidates displayed on the display represents a different person or anobject.
 12. The display method as claimed in claim 11, wherein thesearching includes recognizing the handwritten data, converting therecognized handwritten data into one or more characters, and performingthe search based on the one or more converted characters, and thedisplaying displays the plurality of image candidates obtained from theimage transformation dictionary part by the search.
 13. The displaymethod as claimed in claim 12, wherein the searching includessuccessively converting the handwritten data from the beginning into theone or more characters, and performing the search in the imagetransformation dictionary part by a forward match while increasing thenumber of converted characters every time a character is converted, andthe displaying displays a number of image candidates which decreases asthe number of converted characters used for the search increases. 14.The display method as claimed in claim 11, wherein the displayingdisplays names of the plurality of image candidates together with theplurality of image candidates.
 15. The display method as claimed inclaim 11, further comprising: accepting a selection of one of theplurality of image candidates from a handwritten input part, wherein thedisplaying includes deleting all of the plurality of image candidatesand the handwritten data, and displaying the selected one of theplurality of image candidates accepted by the handwritten input part, ora name of the selected one of the plurality of image candidates, at thesame position on the display as the handwritten data.
 16. Anon-transitory computer-readable recording medium having stored there ina program which, when executed by a computer of a display device, causesthe computer to perform a process including: performing a search for aplurality of image candidates in an image transformation dictionarypart, based on handwritten data; and displaying, on a display, theplurality of image candidates obtained by the search, wherein at least aportion of the plurality of image candidates displayed on the displayrepresents a different person or an object.
 17. The non-transitorycomputer-readable recording medium as claimed in claim 16, wherein theperforming the search includes recognizing the handwritten data,converting the recognized handwritten data into one or more characters,and performing the search based on the one or more converted characters,and the displaying displays the plurality of image candidates obtainedfrom the image transformation dictionary part by the search.
 18. Thenon-transitory computer-readable recording medium as claimed in claim17, wherein the performing the search includes successively convert thehandwritten data from the beginning into the one or more characters, andperforming the search in the image transformation dictionary part by aforward match while increasing the number of converted characters everytime a character is converted, and the displaying displays a number ofimage candidates which decreases as the number of converted charactersused for the search increases.
 19. The non-transitory computer-readablerecording medium as claimed in claim 16, wherein the displaying displaysnames of the plurality of image candidates together with the pluralityof image candidates.
 20. The non-transitory computer-readable recordingmedium as claimed in claim 16, wherein the program causes the computerto perform the process further including: accepting a selection of oneof the plurality of image candidates from a handwritten input part,wherein the displaying includes deleting all of the plurality of imagecandidates and the handwritten data, and displaying the selected one ofthe plurality of image candidates accepted by the handwritten inputpart, or a name of the selected one of the plurality of imagecandidates, at the same position on the display as the handwritten data.